Artificial Neural Network based

Curve Prediction
L ECTU RE C OU RSE : A U SGEW ÄHLTE O PTIM IE RU NGSV ER FA HRE N FÜ R I NGEN IE UR E

S UPE RV IS OR : P ROF . C HRISTIA N H A FN ER

S TU DE NTS : A NTHONY H SIAO , M ICHAE L B OE SCH

Abstract
We use artificial neural networks to perform curve prediction. For that, we have
created a class of neural networks (feed forward multilayer perceptron networks
with backpropagation) that have a topology which is determined by their genetic
makeup. Using a simple evolutionary strategy on their genes, we optimise the
networks’ topologies to solve the problems at hand. Using this approach, we could
generate networks that are able to predict simple functions such as sin(x) or linear
combinations thereof, with moderate computational overhead. However, it was not
possible to generate networks that predict more complex functions such as sinc(x)
or the NASDAQ composite index satisfactorily, within the allowed sizes for the
networks. In general though, it appears to be a useful approach to generate neural
networks using this form of evolutionary strategy as it substitutes for experience in
neural network design.

Introduction
Curve prediction is one of the most popular applications for artificial neural
networks. However, the success of using a neural network to solve a certain
problem is inherently linked to the designer’s ability to apply an appropriate
network to the task. Even relatively simple artificial neural networks such as the
multi-layer-perceptron or variants thereof have several degrees of freedom, e.g. the
number of neurons, the number of hidden layers, the type of transfer functions
employed, which the network is very sensitive to. For most tasks there is no
methodology for designing a neural network which guarantees success. Instead, we
try to evolve a neural network topology that is suitable for any curve prediction
task.

Aim
• To develop an evolvable artificial neural network representation

• To optimise such a neural network to solve a number of curve prediction

 • To evaluate the ability of an evolutionary approach to evolve suitable neural
networks for a given task

Neural Networks
In order to perform the prediction tasks described above, we use multi layer
perceptron networks and a simple backpropagation learning rule. Then, we use an
evolutionary strategy to change the following parameters of the network:

• Number of hidden layers

• Number of neurons in each layer

• Transfer function employed by the neurons in each layer (neurons in the

same layer will employ the same transfer function)

In order to do this, we define a genetic code for the class of neural networks
comprising an N digit binary bit string. In order to limit the optimisation search
space, we arbitrarily limit the number of hidden layers to

The number of neurons per layer is limited to

The allowed transfer functions are linear, linear with bounds and hyperbolic
tangens, as shown below:

Thus, the number of neurons for each hidden layer can be represented by a four bit
number, and the transfer function for the neurons in each layer by a two bit
number, totalling a required six bits per layer. As there are up to 10 hidden layers,
the total bitstring will be 60 bits long, and is represented as follows:

Bit 0 Bit 59
Layer Layer Layer Layer Layer Layer Layer Layer Layer Layer
0 1 2 3 4 5 6 7 8 9

Layer 4, Layer 4, Layer 4, Layer 4, Layer 4, Layer 4,

Neuron bit 3 Neuron bit 2 Neuron bit 1 Neuron bit 0 Transfer bit Transfer bit
1 0

Each layer can have up to 15 neurons, as given by the binary number [Neuron bit 3:
Neuron bit 0]. If the bit string encodes zero neurons for that layer, it is interpreted
as being non-existent. Also, as the bit string can encode for four transfer functions
in each layer, as given by the binary number [Transfer bit 1: Transfer bit 0], but only
three are employed, a bias is given towards the linear y = x transfer function, to be
encoded in two of the possible four states. Thus, by changing the genes of a
network, it will have a different topology, some more others less suitable for the
tasks at hand.

Optimisation
In order to find an optimal network topology for the given tasks, we use an
evolutionary strategy to evolve the genetic makeup of the network, which could
also be regarded as a genetic algorithm without cross breeding.

The Evolutionary Strategy

The algorithm employed works as follows:

1. Produce a first generation of population size seven of random bit strings

2. Generate randomly initialised networks from the population of bit strings

3. Train the networks on a training set using backpropagation

4. Run the networks on the test data

5. A fitness function evaluates the fitness of each network, and the fittest
network is kept for the next iteration, while the other ones are discarded.

6. The fittest network is cloned six times to refill the generation, and each of
these clones is mutated randomly by inverting one of the 60 bits at random.

7. The process repeats at 3. until a maximum number of iterations has been

performed, or a marginal or no improvement in the fitness can be achieved
over several iterations

A table that contains all the bitstrings that have already been evaluated is kept so
as to avoid computing the same network topology multiple times.

Here, we evaluate the fitness of the networks in two ways, depending on the task at
hand.

• For short term prediction, a part of the test signal series is used as input to
the network, and the first value the network predicts is compared with the
 actual value of the series at that point. The cumulative error is found by
summing the absolute difference between the predicted value and the actual
value for all shifted versions of the actual signal as input to the network. The
fitness of the network is the reciprocal of the cumulative error. This method
evaulates the network’s ability to make short term predictions for a given
pattern and number of inputs. (see below)

Actual signal
1 1 . . . . . . . N
.
Neural network with inputs and output
i n p u t s o

2 1 . . . . . . . N
.

i n p u t s o

3 1 . . . . . . . N
.

i n p u t s O

Figure 1: Short term prediction method

• For long term prediction, a part of the signal is used as input to the
network, and the first predicted value is fed back and used as the next input
to the network, and this is repeated for a given number of points that are to
be predicted. Then, the fitness is the reciprocal of the cumulative absolute
difference between the actual signal and the recurrently predicted signal.
This method evaluates the network’s ability to make long term predictions
(forecasts) for a given starting pattern and number of points to predict. (see
below)

Actual signal
1 1 . . . . . . . N
.
Neural network with inputs and output
i n p u t s o

2 1 . . . . . . . N
.

i n p u t s o
3 1 . . . . . . . N
.

i n p u t s O

Figure 2: Long term prediction method

In both cases, there is a minimum error (and thus maximum fitness) that each
network must have, in order to avoid division by zero errors and infinite fitness.
Furthermore, smaller networks, i.e. networks with fewer hidden layers are preferred
over larger ones, as are networks with a small number of neurons.

The Search Space

The size of the search space can be calculated as follows:

In fact, the search space is slightly smaller than this, because some of the
topologies where one or more layers have zero neurons are equivalent. Still, it can
be appreciated, that the search space is large enough to justify this optimisation
approach.

Evaluation and Discussion

We developed a software application with Java © that implements this evolvable
artificial neural network representation, and which allows our evolutionary strategy
to evolve the network topology of the networks. To evaluate our approach, we
adopted the following testing strategy.

Testing Strategy
There are two tasks, short- and long term (function) prediction that our neural
networks will have to perform. Here, we shall qualitatively assess the ability of the
evolved networks to perform each task, using the following representative test
signals:

• Sinusoidal function: An arbitrary sinusoidal function such as Sin(x) with a

given amplitude, frequency and phase. This is probably the simplest test
signal for the networks and the tasks can be expected to be managed
successfully by the networks.

• A-Periodic function: An aperiodic function sinc(x) = sin(x)/x. This is a

challenging function to predict, as it is not periodic, and not monotonous. It
 would come as a positive surprise if the evolved networks would manage the
task successfully.

• Noisy a-periodic or pseudo-random function: An excerpt of the NASDAQ

composite index’ historical weekly values shall be used as an interesting and
challenging test signal. The ability of the networks to predict this stock
market index would be highly surprising and unexpected.

Apart from running both tasks on the four different functions, the following
questions shall be addressed:

• Do we always arrive at the same network for the same problem? In

order to answer this question, several trials of the same test shall be run. If
the same network topologies are arrived at most of the time, it implies that
the evolutionary strategy converges to a local or global optimum, which is
desirable.

• How fit are the networks? As mentioned above, there exists a maximum
fitness that a network may achieve. How fit, relative to the maximum
achievable fitness, are the evolved networks?

Tabular Summary
The section below describes important aspects of the individual tests in detail. In
addition, the table below summarises the results.

Criteria – 5 Sin(x) Sinc(x) NASDAQ

inputs
Short term   
prediction
Long term   
prediction
Same   
networks
Fitness 17% 15% 2%
%-Error 36% 86% 48%
Convergence   

Table 1: Summary of the tests for 5 inputs

Criteria – Sin(x) Sinc(x) NASDAQ

10 inputs
Short term   
prediction
Long term   
prediction
Same   
networks
Fitness 95% 15% 2%
%-Error 0% 113% 44%
Convergence   
Table 2: Summary of the tests for 10 inputs

Criteria – Sin(x) Sinc(x) NASDAQ

20 inputs
Short term   
prediction
Long term   
prediction
Same   
networks
Fitness 99% 15% 3%
%-Error 0% 95% 27%
Convergence   
Table 3: Summary of the tests for 20 inputs

Test Details
Sinusoidal
The neural networks evolved are able to predict the sinusoidal signals with
acceptable accuracy, provided they receive enough inputs. Figure 3 below
illustrates the evolution process over several trials. Each point on the graph
represents an improved network topology over the previous one.

Figure 3: Fitness evolution for different trials for Sin(x) - Clear fitness improvement

As with the fitness evolution, the error performance of the evolved networks
improves. Figure 4 below illustrates how the %-error in the long term prediction
decreases in general with each generation.
Figure 4: Error evolution for different trials for Sin(X) - Clear performance improvement

Comparing the size of the network (number of neurons inside hidden layers) to the
network’s performance, it appears, that there exists a certain range of ‘right’ sizes
that the network should have, which allows it to achieve high fitness. Another way
of looking at this is that the network should have a certain minimum complexity (in
terms of numbers of neurons) which is adequate to solve the task at hand. Below
that critical size, it is unlikely, that a network can achieve a high fitness.

Figure 5: Size does matter – a network needs a certain minimum size or complexity to
achieve high fitness
 Sinc(x)
Unlike in the previous case with sin(x), the evolutionary approach does not generate
sufficiently fit networks to perform long-term prediction on a sinc(x) function. Figure
6 and Figure 7 outline the evolutionary performances over several trials. They show
clearly, that the evolutionary approach works in principal, i.e .networks are evolving
and improving, however the task to predict a sinc(x) seems to be too ‘difficult’ a
task for the simple feed-forward perceptrons employed here. It appears that the
network evolution is hitting a fitness and %-error performance limit at about 15%
(f)and 90%(e) respectively.

Figure 6: Fitness evolution for different trials for Sinc(x) - No clear fitness improvement
Figure 7: Error evolution for different trials for Sinc(x) – The error decreases, but is still
unacceptably high

NASDAQ
Attempting to perform long/term prediction on the NASDAQ is ambitious. Here, the
evolutionary approach again works somewhat, as it is able to generate networks
with improving performance over several generations, however the network model
or complexity is again not able to cope with the challenge posed by the NASDAQ.
Figure 8 and Figure 9 summarize the network evolution over several trials.
Figure 8: Fitness evolution for different trials for the NASDAQ - Too difficult for the
networks
Figure 9: Error evolution for different trials for the NASDAQ - Error performance improves,
but it is still too high

General Comments
• The networks’ performances are closely tied to some randomness in
the initialisation and the success of the training. The training method
employed, backpropagation, however is not guaranteed to achieve a
satisfactory level of training, and does not necessarily find the globally
optimal solution parameters for the network. To overcome this, we repeated
the training sequence of the networks several times to increase the likelihood
of obtaining a ‘’well trained’ network. This however increased the
computational load manifold, to an impractical degree, and is therefore not
an adequate remedy to decouple the success of the evolutionary approach
from its sensitivity on randomness and initial conditions.

Conclusion
• To develop an evolvable artificial neural network representation

• To optimise such a neural network to solve a number of curve prediction

• To evaluate the ability of an evolutionary approach to evolve suitable neural

networks for a given task

The performance of an artificial neural network for a given problem is inherently

tied to the topology of the network, and it might prove difficult to produce
satisfactory results. Thus, we have set out to create a neural network design
framework that replaces the designer’s experience with an evolutionary strategy.

We have developed a bit string representation that corresponds to the genetic

makeup of a multi-layer perceptron network, allowing an optimisation of the
following network parameters: number of hidden layers, number of neurons in the
hidden layers and the transfer function employed by the neurons in each layer.

Using a simple evolutionary strategy, we tried to optimise the network topology for
the tasks at hand by optimising a generation of networks’ genetic makeup. In
principal, this approach has proven valid, and we have demonstrated an evolution
of networks to predict a sinus function. For more complex function such as the
sinc(x) function or the NASADQ, the evolutionary approach worked, although it was
limited by the network model’s inherent ability to predict complex patterns.

We are optimistic about the approach to evolve neural network topologies for given
tasks, and there are several aspects that could be improved or further investigated.
In particular, we suggest the following:

• Within our simulations, we had to limit the search space by limiting the
allowed size of the networks and their transfer functions. A more extensive
investigation into the evolutionary approach could include other non-linear
transfer functions, larger networks as well as more interconnected or feed-
back networks.

• A major limiting computational factor was the learning rule employed

(backpropagation). It would be worthwhile to consider other learning rules.

• In our approach, we ‘allow’ evolution to uniformly develop networks of all size

and shapes. It might be a more fruitful approach to instead constrain the
evolution to ‘organic growth’, i.e. to start with a small network and growing
them by evolution.

• We have used our evolutionary strategy on function prediction only. It would

be worthwhile to apply this method to other common aNN tasks, such as
image recognition.