Digital Implementation of Artificial Neural Networks:

From VHDL Description to FPGA Implementation

N. lzeboudjen, A. Farah*, S. Titri, H. Boumeridja

Development Center of Advanced Technologies, Microelectronic Laboratory
128, Chemin Mohamed Gacem, B.P: 245 El Madania, 16075 Algiers-Algeria.
E-mail: nizeboudjen@hotmail.com
Fax: 213 02 27 59 37
*Ecole Nationale Polytechnique, Laboratoire Teclmiques Digitales et Systemes.
10, Avenue Hassen Badi E1 Harrach, Algiers- Algeria.
E-mail: farah@ist.cerist.dz

Abstract
This paper deals with a top down design methodology of an artificial neural network
(ANN) based upon parametric VHDL description of the network. To come off early in the
design process a high regular architecture was achieved. Then, the VHDL parametric
description of the network was realized. The description has the advantage of being
generic, flexible and can be easily changed at the user demand. To validate our approach,
an ANN for electroc,'u'diogram (ECG) arrhythmia's classification is passed through a
synthesis tool, GALILEO, for FPGA implementation.

Key words
ANN, top down design, VHDL, parametric description, FPGA implementation.

Introduction
Engineers have long been fascinated by how efficient and how fast biological neural
networks are capable of performing complex tasks such as recognition. Such networks are
capable of recognizing inputs data from any of the five senses with the necessary
accuracy and speed to allow living creature to survive. Machines, which perform such
complex tasks, with similar accuracy and speed, were difficult to implement until the
technological advances of VLSI circuits and systems in the late 1980's [I]. Since then,
VLSI implementation of artificial neural networks (ANNs) has witnessed an exponential
growth. Today, ANNs are available as microelectronics components.
The benefit of using such implementation is well described in a paper by R. Lippman [2] :
<< The great interest of building neural networks remains in the high speed processing that
could be provided through massively parallel implementation >>. In [3], P. Trealeven and
others have also reported that the important design issues of VLSI ANNs are parallelism,
performance, flexibility and their relational ship to silicon area. To cope with these
properties [3] reported that a good VLSI ANN should exhibit the following architectural
properties:
9 Design simplicity that leads to ,architecture based on copies of a few simple cells.
9 Regularity of the structure that reduces wiring
9 Expandability and design scalability that allow many identical units by packing a
number of processing units on a chip and interconnecting many chips for a complete
system.
Historically, the development of VLSI implementation of artificial neural networks has
been widely influenced by the development in technology as well as in VLSI CAD tools.
 140

Hardware implementation of ANNs can make use of analog or digital technology

techniques. A straight question is how to choose between these two technologies?
Selection between digital or analog implementation depends on many factors such as
speed, precision, flexibility, progranunability and memory elements.
Analog implementations have the potential for high densities and fast operations.
Unlbrtunately, they are sensitive to noise; cross talk, temperature effects and power
supply variations. Also long term weight storage requires special fabrication techniques.
Another major drawback, which is very critical in ANNs is that conventional analog
implementations are fixed (i.e. no programmability can be achieved).
Digital integrated technology, in the other hand, offers very desirable features such as
design flexibility, learning, expandable size and precision. Another advantage is that
mature and powerful CAD tools support design of digital VLSI circuits.
Digital implementation of ANNs can make use of full custom VLSI, semi custom, ASICs
(application specific integrated circuits) and FPGAs (Field programmable gate arrays) [4],
[5], [6].
Particularly, FPGA implementation of ANNs is very attractive because of the high
flexibility that can be achieved through the re-programmability nature of these circuits.
One would assume that the neural network models developed in computational
neuroscience could be directly implemented in silicon. This assumption is false because
when implementing a neural network, the designer is confined to some specific problems
related to the characteristics of these algorithms such as: speed processing, precision, high
memorization, parallelism, regularity and flexibility of the architecture. In addition, the
designer must fulfil design constraints related to the target application: area and
consumption problems. Another supplementary imperative constraint which adds, today,
to the complexity of these circuits is the quick turnaround design.
Nowadays, with the increasing complexity of VLSI circuits, state of the art design is
focused around high level synthesis which is a top down design methodology, that
transform an abstract level such as the VHDL language (acronym for Very high speed
integrated circuits Hardware Description Language) into a physical implementation level
[7], [8], [9].
VHDL based synthesis tools have become very popular due to mainly these reasons: the
need to get a correctly working system the first time, technology independent design,
design reusability, the ability to experiment with several alternatives of the design, and
economic factors such as time to market. In addition, synthesis tools allow designers with
limited knowledge, of low level implementation details to analyze and trade off between
alternative implementations without actually implementing the target architecture [9].
Beside this, the VHDL language is well suited for high regular structures like neural
networks.
However, although all these advantages, seldom attention has been done to use synthesis
for ANNs implementations.
In this paper, a new design methodology of ANNs based upon a VHDL synthesis of the
network is applied. The novelty is the introduction of the parametric VHDL description of
the network.
The intended objective is to realize an architecture that takes into account the parallelism,
performance, flexibility and their relational-ship to silicon area as requested in [3]. After
synthesis the resulting netlist file is mapped into the FPGA X1LINX XC4000E family
circuit's for physical implementation [10]. The paper is organized as follow:
In section II theoretical background of artificial neural networks is given. Section III
describes the followed design methodology. In section IV, Parametric VHDL description
 141

of the ANN is introduced. Section V is an application to an ECG arrhythmia's classifier.

Finally, a discussion and conclusion are given in section VI.

II. Theoretical background

An artificial neural network (ANN) is a computing system that combines a network of
highly interconnected processing elements (PEs) or neurons (Fig.l). Connections between
neurons are called synapses or connection weights.
hlspired by tile physiology of the human brain, the traditional view holds that a neuron
performs simple threshold function- weighted input signals are assumed-. If the results
exceed a certain threshold, a signal emits from the neuron. Fig. 1. (a). Represents a
biological model neuron and Fig. 1. (b) represents an artificial neuron model.
Many different types of ANNs exist: single layer perceptron, multilayer perceptron, the
Hopfield net and Boltzman machine, the Hamming net, the Carpenter/Grossberg classifier
and Kohenen's self-organizing maps [2]. Each type of ANN exhibits its own architecture
(topology) and learning algorithm. From all these types of ANNs we have chosen for
implementation the three layer feed-forward back propagation network (Fig.l.c). This
choice is motivated by the high regular structure, the simple connection (unidirectional)
and the great number of problems that can be solved by this kind of neural networks
ranging from classification, pattern recognition and image processing to robotics and
control applications.

i~put~ weights Activation

function

Xn , ~ L J l

Synapses Output layer

Hidden layer
(a) (b) Input layer
Synaptic connection
O Processing element (PE) or neuron

(c)

Fig. 1. (a) Biological model neuron. (b) Artificial neuron model (c) Three layer artificial neural network

The ANN computation can be divided in two phases: learning phase and recall phase. The
learning phase performs an iterative updating of the synaptic weights based upon the error
back-propagation algorithm [2]. It teaches the ANN to produce the desired output for a set
of input patterns. The recall phase computes the activation values of the neurons from the
output layer according to the weighted values (computed in the learning phase).
Mathematically, the function of the processing elements can be expressed as:
l j = )-~ ,wiji ~ail - l ) + O (I)
i
w.[ is the real valued synaptic weight between element i in layer l-1 and element j in layer
U
O-l)
I. s i is the current state of element I in layer I-I. 0 is the bias value. The current state
 142

of the node is determined by applying the activation function to x I . For our

implementation, we have selected the logistic activation function:
sl - 1 (2)
I
I + e -xi
Training (learning) of an ANN is carried as follows:
i) Initialize the weights and bias,
ii) Compute the weighted sum of all processing elements from the input to output layer,
iii) Starting from the output layer and going back wordto the input layer adjust the
weights and bias recursively until the weights are stabilized.

It must be mentioned that our aim is to implement the recall phase of a neural network,
which has been previously trained on a standard digital computer where the final synaptic
weights are obtained, i.e. "off- chip training".

III. D e s i g n methodology
The proposed approach for the ANN implementation follows a top down design
methodology. As illustrated in Fig. 2, architecture is first fixed for the ANN. This phase
is followed by the VHDL description of the network at the register transfer level (RTL)
[8], [13], Then this VHDL code is passed through a synthesis tool which performs logic
synthesis and optimization according to the target technology. The result is a netlist ready
for place and root using an automatic FPGA place and root tool. At this level verification
is required before final FPGA implementation.

Fig.2 Design methodology of the ANN

In the following sections the digital architecture of the ANN will be derived then the
proposed parametric VHDL description. Synthesis results, placement and rooting will be
discussed through an application.
 143

lIl Digital architecture of the ANN

As mentioned in section I, the requirements of ANNs are parallelism, performance,
flexibility and their relational-ship to silicon area (in our case number of CLBs).
Parallelism of the network is discussed in this section.
Designing a fully parallel ANN requires:
9 The parallelism of all layers, which means that at least OnE multiplier, is needed per
layer.
9 The (PE) or neuron's parallelism which requires one multiplier per neuron.
9 The connection parallelism, which means that all synaptic connections of a neuron,
are calculated at the same time. In this case, the neuron needs as many multipliers as it
has connections to the previous layer.
9 The connection parallelism is the highest degree of parallelism that can be reached in
an ANN. This parallelism leads to a very high network performance in term of
processing speed. However building a large number of multipliers and a large number
of connections are a severe penalty for the FPGAs because of their limited resources
and the excessive delay inherent to FPGA. To avoid this problem, we consider only
the neuron's parallelism. ConsequEntly, data transfer between layers should be serial,
because one neuron is chosen to compute only one connection at a time.
Based upon the above ANN hardware requirements, the FPGA equivalent architectural
model of the neuron of Fig. l.b is represented by Fig. 3.a. The hardware model is mainly
based on a:
9 Memory circuit (ROM) where the final values of the synaptic weights are stocked,
9 Multiply accumulate circuit (MAC) which computes the weighted sum and,
9 Look-up table (LUT) which implements the sigmoid activation function.
ThE resulting ANN architecture of Fig. l.c. is represented in Fig. 3.b. (note that only the
second and output layers are represented in this figure), with the following features:
9 For the same neuron, only one MAC is used to compute the product sum.
9 Each MAC has its own ROM of weights. The depth of each ROM is equal to the
number of nodes constituting its input layer.
9 For the same layer, neurons are computed in parallel.
9 Computation between layers is done serially.
9 The whole network is controlled by a unit control.
As we can see, in Fig. 3.b., the resulting architecture exhibits a high degree of parallelism,
simplicity, regularity and repeatncss.

Fig. 3. (a): Neuron hardware model. (b) ANN architecture.

 144

IV. Parametric VHDL description of tile ANN

Having fixed architecture, the next phase is the VHDL description of the ANN.
Flexibility is the parameter of interest in this section.
The capabilities of the VHDL language to support parameterized design are the key to
providing flexible ANNs that can adapt to different specifications.
Besides its emergence as an industry standard for hardware design VHDL supports
additional features such as encapsulation, inheritance, and reuse within the representation
[Ill.
Encapsulation reduces the number of details that a designer has to deal with, through the
representation of the design as a set of interacting cores. Thus the designer doesn't have
to know how these cores work inside, but rather should focus on defining the appropriate
interfaces between them. Encapsulation is reinforced through the use of packages,
functions, procedures and entity declaration.
Inheritance in VHDL is realized through parameter passing. The general structure of
components and architectures is inherited by new designs. The parameters are passed to
instantiate the design to the specifications of the target application. Inheritance is also
reinforced through component instantiation. Reuse can be realized by constructing
parameterized libraries of cores, macro-cells and packages.
Our approach to the ANN hierarchic VHDL description is illustrated in Fig.4. VHDL
description of the network begins by creating a component neuron, then a component
layer is created and finally a Network is described.
9 Component neuron is composed by a MAC component, a ROM component and a LUT
component.
9 Component layer is composed by a set of component neurons and multiplexers.
9 A Network is composed by a set of component layer (input layer, hidden layer and
output layer).

Fig. 4 Top view of an artificial neural network

In Fig. 5(a) the VHDL description of the neuron is illustrated. Fig. 5(b) illustrates the
layer description. Fig. 5(c) illustrates the Network description.
First, a VHDL description of the MAC circuit, the ROM and the LUT memories was
done. In other to achieve flexibility, the word size width (nb_bits) and the memories
depth (nb_addr and n b a d d ) are kept as generic parameters (Fig. 5(a)).
Second, a VHDL description of the neuron was achieved. The parameters that introduce
the flexibility of the neuron are the word size (rib_bit) and the component instantiation. A
designer can change the performances of the neuron by choosing other pre-described
components stocked in a library without changing the VHDL code description of the
neuron (Fig. (5b)).
 145

Third, a layer is described. The parameters that introduce the design flexibility and
genericity of the layer are the word size ( n b b i t s ) and the number of neuron (nb_neuron).
The designer can easily modify the number of neurons in a layer only by easy
modifications of the layer VHDL code description (Fig.5. b.).
Finally, a VHDL description of the network is achieved. The parameters that introduce
the flexibility of the network are the neurons word sizes (n), the number of neurons in
each layer (nb_neuron) and component instantiation of each layer (component layerS,
component layer3 and component layer2). The designer can easily modify the size of the
network simply by giving small changes in the layers descriptions. The designer can also
change the performances of the network only by using others pre-designed layers (Fig
4.c.).
entity neuron is Entity nelwock is
generic(nb_bits :integer) ; -- word size generic (n, nl, nO: integer) ;
port (in nenr :in unsigned(nb_bits- I downto O) ; pelt (X I,X2,X3,X4,X5:in sl.d_logic_vector (n downto 0);
out_neur : out std_logic_v~tor((nb_bits - L) downlo O) ; ad:in unsigned(nl dowmo 0);
rend_en,rst,clk,rcady : in std_logic) ; adl:in unsigned (nl dew, ate 0);
end neuron ; ad2:in unsigned(hi downto 0):
,lrchileclnrc nenron_dc,~tiplion of neuron is elk ,rst.rendyl.rend en : in ~d logic;
zompoucm M A C c 132,e232:oet std_logic_vector(((2 *n+ 1) downto 0)) ;
generic (nb_bits : integer) ; end network ;
port (x, w : i n std_logic_vcctor((nb_bits-I) downto 0) ;
elk. rsl : in sld_logic ; architecture network_desctiplion of nctwoA is
q : out std_logic_vector ((2*nb bits) -I) downto 0)) ; component layer I
gild cotnponen[ ; generic (nb_neeron : integex ; nl : integer) ;
component ROM port(XI,X2,X3.X4,X5:in std_h~ic_vcctor (nl downlo 0);
generic {nb add : integer : ni',_bits :integer) ; ad:in unsigned(nl downto0).
port ( add : in unsigl~'d ((nb_addr -1) downto 0) ; s l:in std logic_vector(no downlo 0);
out_tom : out ~d logic vector((nb_bits - I) downto 0) ; clk,rsl.rendy,read_en : in std_logic) ;
read en : in ~d_logic) ; n 13.n23,n33,n43.n53 :out std logic_vector(((2*n l )+ I )
end colnponeflt ; dowmo 0));
[,'OIllpone n.I LUT end component ;
generic(nb_ad(h" :integer ; nb_bits :integer) ;
port (addr : in tad Iogie_vectoc((nb_bits - I) downto 0)); component layer2
out lut : out sld_logic_vector((2*nb bits -I) downto 0) ; genetic (nb_ncuron : integer ; nl : integer) ;
read en : in std_logic) ; port(X l,X2,X3,X4,X5:in s~d_togic_vector (at downto 0);
end component ; adl :in unsigned(nl dowmo 0);
begin s2:in sld_logic_vector(nO downto 0).
rein_wight : ROM generic inap (). port nrmp (read en, add, w) ; clk,rsl,ready,.rend_en : in std logic) ;
molt ace : MAC genetic map(), port map (x.w,clk.rea,q) ; n 13.n23,n33:out std Iogic_veCter(((2 *n I)+ 1) dowmo
result : LUT generic map O . port map (rend_en. q. out_lut ); 0));
end neuron_de~ription ; end component ;
(a)
component layer3
entity Inyer_n is genetic (nb_neuron : integer ; n I : integer;) ;
generic(nb neuron :integer ; rib_bits :imeger) ; port(X ~,X2,X3:in std_loglc_vector {n ~ downlo 0);
p~t(inpot_layer I : unsigned ((nb_bits -I) downto 0); nd2;in unsigned(nl dowmo 0);
inl~tt Inyer2 : in ~td Iogic_vector((nh hits) downlo 0); s3:in md logic vector(no (k)wnlo 0);
elk, rst, ready.rcad_enl : :in ~ d [t~ic ; clk,r~,ready,rend cn : in ~d logic) :
output_layer I ..... output layer n : out n 132,n232 :out sl d_logic_vector(((2 *n l )+ I ) downl o 0));
~d..iogic((2*(nb_bits)+ I) downto 0)) ; end component ;
end layer_n ;
architecture layer_description of layer n is ~r
component neuron layer 5 : laycri genetic n'mpO, port mal~sl, X[,X2,X3,X4,X5,
port (in_neur :std_iogic_vector(nb bits- I downto 0) ; rs~,clk, ready.read_cn,ad,n 13.n23,n33,n43,n53) ;
out_neur : out sld_loglc_vcctor(nb_bits - I dowmo O) ;
read_en.rst.clk.ready : in std_logic) ; layer 3:layer2 generic innpO, Portrnap(s2.X I,X2,X3,X4,X4,
end eounponent ; elk jsl,ready,read en,ad I .n ! 3,n23,n33);
begin layer_2:layer3 genetic map(), port nmp(s3,X l,X2,X3,clk,r~,
neuron_n : neuron generic map(), rendy.read_en,nd2,n 132.n232);
port map (input_laycxi ,input_layer2. elk, rst. ready, :nd ;
read_enl .output_layerl....,outpuLlayer n) ;
end iayer_descriiXion;
(b) (c)

Fig. 5. Parametric VHDL description. (a): Neuron description. (b): Layer description. (c): Network
description.
 146

V. Case Study: ANN A r r h y t h m i a ' s classifier synthesis

V.1 Classifier description

To validate our approach, our first application is an electrocardiogram (ECG) neural
network classifier used to distinguish between normal sinus rhythm (NS) and the
supraventricular tachycardia (SVT).
The system is composed of two cascaded stages: a morphological classifier and a
temporal classifier (Fig. 6.). The morphological classifier is designed to distinguish
between normal (N) and abnormal (A) P and QRS waves patterns of the ECG signal,
respectively. The temporal classifier takes the first stage outputs and the PP, PR, RR
interval duration of the ECG's signal rhythm and outputs a classification in two
categories: NS or SVT [12]. First a program was written in C code to train the two
networks using the back propagation learning algorithm, and where the final weight are
obtained. After training only the timing classifier, which is composed of an input layer of
5 neurons, a hidden layer of 3 neurons and an output layer of 2 neurons i.e. (5-3-2)
classifier, was synthesized for FPGA hardware implementation.

NS

~swr

PR
PP
ECG ~iB.nl

Fig. 6. Neural network arrhythmia's classifier architecture

V.2 Synthesis and simulation results

For IC design, the architecture was synthesized using the synthesis tool GALILEO [13].
Before synthesis simulation is required until the ANN meets the functional specifications.
Fig.7.a input-output pins of the (5-3-2) classifier. For this application, the data word size
is fixed to 8bits. As precision is not requested, a scaling was done in order to reduce the
sizes of the look-up tables in each layer. Thus the network outputs are 8 bits size.
Fig.7.b represents functional simulation results of the ANN. Results show that the
required functionality is well achieved.
Once the functionality is verified, the VHDL - RTL code is used for synthesis. At this
level, and depending on the target technology, which is in our case, the FPGA Xilinx,
GALILEO transforms the RTL description to a netlist in term of configurable logic
blocks (CLB). Tile synthesis tool proceeds to estimate area in terms of CLBs. The output
of GALILEO is a table summarizing synthesis results from individual passes as well as
the best result of the (5-3-2) network based on the desired performance optimization
(speed/area). In this application, we selected area for optimization because the ECG
signal is slow (0.8 sec per cycle).
Fig.8. Shows synthesis results of the (5-3-2) network with the XC4000E as target
technology. In addition, Galileo outputs a netlist file (xnf file format) which will be used
in the next phase for placement and routing.
 147

Fig. 7. (a): ANN input- output connections. (b): Functional simulation results of the (5-3-2) ANN.

V.3 FPGA implementation

This phase deals with the automatic placement and rooting using the XACT tool.
At this level, XACT takes as input the netlist file (xnf format) generated by GALILEO.
The resulting structure of the ANN is shown in Fig. 9. The ANN is mapped into the
4020EPG223 package. For clarity, only the placed CLBs are shown in Fig.9. As we can
see the (5-3-2) network is mapped into only one FPGA.

Fig. 8. Galileo Synthesisresults. Fig.9. Top view of the ANN FPGA structure.
 148

V.i Discussion and Conclusion

Through this paper we have presented a synthesis methodology for FPGA implementation
of a digital ANN classifier.
The proposed VHDL description is based on a simple, regular and parallel architecture.
The use of the parametric VHDL description offers a high flexibility to the designer
because the same code can be reused to cover a wide range of applications and
performances depending on the pre-designed ANN library.
In addition, the advantage of using synthesis is that the designer can target the circuit for
different libraries (XC3000, XC4000, XC700, Actel ACT2, MAX5000, and ASICs) from
different vendors (Xilinx, Actel, Altera, etc.). After comparing, the designer can choose
the best technology that meets the requested input specifications.
The primary results are very successful since the whole network can be mapped into only
one FPGA.
Our next objective is to test the FPGA circuit in the whole ECG system. In the future, our
objective is to include the training phase in the proposed architecture (on chip training), to
explore tile proposed ANN description to other applications domains (image processing,
character recognition, etc.) and to extend the approach to other ANN algorithms
(Hopfield, Kohenen...etc.).

References
[1] M. I. Elmasry, <<VLSI Artificial Neural Networks Engineering >~,Kluwer Academic
Publshers.
[2] Richard P. Lippmann, ~An Introduction to computing with Neural Nets ~, IEEE
ASSP Magazine, pp. 4 -22. April 1987.
[3] Philip Trealeven, Macro Pacheco and Marley Vellasco, ~ VLSI Architectures for
Neural Networks ~, IEEE MICRO, pp. 8-27, December 1989.
[4] Y. Arima, K. Mashiko, K. Okada, ~A Self- Learning Neural Network Chip with 125
Neurons and 10K Self-Ornization Synapses ~, Symposium on VLSI Circuits, pp. 63-
64, 1990 IEEE.
[5] H. Ossoing, ~Design and FPGA- Implementation of Neural Networked, ICSPAT'96,
Pp. 939-943.
[6] Charles E. Cox and W. Ekkehard Blanz, ~ GANGLION- A Fast Field Programmable
Gate Array Implementation of a Connectionist Classifier ~, IEEE JSSC, Vol. 27, No.
3, pp. 288- 299, March 1992.
[7] R. Airiau, J. M. Bcrge, V. Olive, J. Rouillard " VHDL du language a la
modelisation",
Presses Polytechniques et Universitaires Romandes et CNEST- ENST.
[8] R. Airiau, J. M. Berge, V. Olive, "Circuit Synthesis with VHDL", Kluwer Academic
Publishers.
[9] Daniel Gajski, Nikil Dutt, Allen Wu, Steve Lin, "High level Synthesis- Introduction
to
Chip and System Design", Kluwer Academic Publishers.
[10] XACT user manual.
[11] M. S. Ben Romdhane, V. K. Madissetti and J. W. Hines, " Quick-Turnaround ASIC
Design in VHDL Core- Based Behavioral Synthesis", Kluwer Academic Publishers.
[12] N. Izcboudjen and A. Farah, " A New Neural Network System for arrhythmia's
Classification," NC'98, International ICSC/IFAC Symposium on neural
Computation. Vienna, September 23-25, pp. 208-212.
[13] GALILEO HDL Synthesis Manual.