&VSPQFBO
$POUSPM
$POGFSFODF

&$$



"VHVTU
o

4FQUFNCFS

,BSMTSVIF
(FSNBOZ

$1HXUDO3UHGLFWLRQ0RGHOIRU0RQLWRULQJDQG)DXOW'LDJQRVLVRI
D3ODVWLF,QMHFWLRQ0RXOGLQJ3URFHVV
1&RVWD%5LEHLUR

CISUC – Centro de Informática e Sistemas da Universidade de Coimbra

Department of Engineering Informatics, Coimbra University, Portugal
Fax: +351 39 701266 Telephone: +351 39 790000
e-mail: nfilipe@eden.dei.uc.pt, bribeiro@eden.dei.uc.pt

.H\ZRUGV Industrial Intelligent Control, Principal
Component Analysis, Neural Networks, Prediction.
$EVWUDFW
In engineering systems, early detection of the occurrence of
faults is critical in avoiding product defects. This problematic
is here discussed in the framework of an industrial process,
namely, an injection moulding plastic machine. The
relationships between the process state and the product
quality are achieved through Principal Component Analysis.
After having identified the main variables, two neural
network architectures were investigated, TDNN and Elman
networks, with respect to one-step ahead prediction. The
results show that TDNN exhibited lower training times with
respect to a desired performance criteria. However, for time
series in which temporal dependency is large, the recurrent
networks with time delayed inputs could lead to better results.
,QWURGXFWLRQ
Neural networks have achieved, in recent years, a high degree
of importance. The availability of process control computers
as well as data historians made it easy to develop neural
network solutions for process modeling and control. Their
usefullness is no longer questioned. From medical to
industrial applications, neural networks have been applied in
a countless number of situations. Lately, mainly due to the
increasing pressure from the consumption markets, a lot of
research has been done under the field of fault detection and
diagnosis. Such systems, shortly refered to as FDD systems,
have become extremely important among industrial
applications. The needs to produce more and better, imposed
by the more demanding and better informed markets, has led
to the investigation of new methodologies in the quality
control area. Neural computing, as one of such techniques,
became an attractive approach in this area, since neural
networks are adaptable to an evolving environment, and are
able to take a quick decision once they have learned the
proper control function.
Artificial neural networks (ANN), because they are cost-
effective, easy to understand and because of their ability to
learn from examples, have found many applications in
process modeling and control, as intelligent sensors, to
estimate variables that usually cannot be measured on-line, in
dynamic system identification, in fault detection and
diagnosis and, finally, in process control.
This paper addresses the use of artificial neural networks
in process modeling. Namely, in a plastics injection moulding
industrial process. In section 2, a description of the process is
given. In section 3, neural data prediction is discussed.
Following the need to understand the process behaviour and
how it responds to changes in the environment, the data
processing analysis is performed in section 4. Finally, in
section 5, an industrial intelligent system based on neural
networks is proposed and the obtained results displayed.
,QMHFWLRQPRXOGLQJSURFHVVGHVFULSWLRQ
In principle, injection moulding is a simple process. A
thermoplastic, in the form of granules, passes from a feed
hopper into the barrel where it is heated and melted. It is then
injected into a cold mould, which is clamped tightly closed.
When the plastic has had enough time to return to a solid state
the mould opens and the produced part is extracted. All
details of the mould are reproduced in the finished part.
Although the injection moulding process can be applied to
a large variety of polymers, the manufacture of only one
particular grade is considered here. The part, from the
automotive industry, was moulded on a DN502 mould and
using the grade Hostacom D M2 T06.
)LJ Injection moulding machine.
The process variables for which, in the current mould,
data is available are cycle time, dosage time, injection time
and cushion. The last two variables can be seen in figure 2.

*4#/


 InjectionTime
 3URGXFW



             
Cushion




 relationships are generally unknown and not easily identified.
             
)LJ Industrial plastic injection process data for a set of 782
samples.
1HXUDOGDWDSUHGLFWLRQ
One of the most promising current applications of neural
networks is the development of intelligent sensors. Often, the
important variables in an industrial process are not available
during the production run, but with some nonlinear statistics
it is possible to associate them with the available
measurements.
In the injection moulding process, 10 to 20 variables are
usually available. However, even though only a few number
of them are truly significant, the existence of strong
nonlinearities and interdependencies between them, increases
the complexity of the process. The prediction of such key
process variables is therefore a crucial task.
Neural networks are known for being able to efficiently
approximate and interpolate multivariate data. Their ability to
deal with nonlinearities and their capability to generalize from
the acquired knowledge during the training phase, makes
them extremely suitable for prediction.
Until recently, quality control was performed off-line in
some kind of quality control laboratory, as shown in Figure 3.
Obviously, a time delay is incurred in analysing the test
samples, which can be in the range of some minutes to some
hours. This could be too late to make the necessary
adjustments.
One can go even further in terms of prediction and look
into an important aspect such as the one concerned with
process control. Poor quality parts are usually associated with
some deviation in the variable values. If the system could
predict in advance the process behaviour, it would be able to
identify future deviations and proceed attemptly with the
necessary corrections.
3URFHVV
,QSXW
3URFHVV /DE7HVW
4XDOLW\
)LJ Conventional quality control procedure.
3ULQFLSDO&RPSRQHQW$QDO\VLV
In order to identify the relationships between variables and to
detect the possible causes of faulty situations, some statistics
have been used. By examining the way the system reacts to
variations in the process variables, it is possible to identify the
most significant parameters related with product quality, and
to detect the causes for process anomalies. However, these
The relationships between the process state and the product
quality can be statistically described either by conventional
approaches or Principal Component Analysis.
Common methods are useful to determine the
relationships between process variables, by examining the
covariance matrices or the correlation coefficients. High
coefficients clearly show strong correlation between the
parameters. However, in the presence of time dependencies or
non-linear relationships, the analysis will fail, since it only
works for linear time-independent processes. Nevertheless,
although the injection moulding process is known to be
extremely non-linear, some variables seem to be related, as
indicated by the relatively high calculated correlation
coefficients. Alternatively, Principal Component Analysis
allows clusters of linked variables to be derived. One may
think on PCA as a technique to find the directions in which a
cloud of data points is most stretched. In this way, it is
possible to extract the main relations in data as well as to
detect its abnormal behaviour. When the process is moving
out-of-control variables indicative of the cause of the
deviation must be identified. Examining their contribution to
the calculated scores, it is possible to identify the set of
original variables whose contribution may be reflective of the
non-conforming behaviour.
Figures 4 and 5 show the scores plots for principal
components one and two and principal components three and
four, respectively ( '+' define the nominal data, whilst 'o'
define abnormal data). As can be seen, the evidence of a
problem is most clearly highlighted by principal component
four. The identification of the combination of variables,
which contribute to the particular principal component of
interest (pc4), allows the identification of the most probable
causes of the faults. In our case, cushion and injection time
were the ones that contributed the most to the pc4.

 6&25(63/27 A neural network is trained on historical data, obtained

from the injection moulding machine during running

operation, to predict process variables one step ahead. We can
thus construct a neural model of the real process. The main
objective is the detection of defects on the produced parts as
3ULQFLSDO&RPSRQHQW



well as the identification of its causes. Parts with burning

 marks, spots, streaks or anomalous dimensions should be
detected and isolated. Once the faults are known and its

causes identified, one can easily train a neural network to act

on the system's responses in order to detect anomalies and
identify its causes. Figure 6 shows the overall proposed
 scheme.
Due to the dynamic behaviour of the process, if we

        assume that the input variables xt are samples of a time
3ULQFLSDO&RPSRQHQW
series, and a future value is to be predicted, then the neural
)LJ Scores plot for principal components 1 and 2 with network becomes dynamic. This leads to another problem.
process data. The one of finding structure in a time varying data sequence.
There are basically two different approaches. The simpliest
6&25(63/27
 one, known as time lagging or temporal windowing, is to
convert a time varying signal into a static vector. The

samples x1 , K , x n can be stored in a delay line, which
serves as the input layer to a feedforward multi-layer
3ULQFLSDO&RPSRQHQW

perceptron (MLP) network. As each new data point becomes

 available, the data is shifted along to accommodate it [Swi1].
This can be regarded as a moving window which covers the

last n values in the sequence. Networks that use data in this

way are known as TimeDelayNeuralNetworks, TDNN.
Another method, for representing time, is to feed a
 recurrent network with single values in a sequence and allow
it to learn its own short time memory. Elman networks have

      
been used in this work. These networks have a multi-layered
3ULQFLSDO&RPSRQHQW structure similar to the one of MLPs. The main difference is
)LJ Scores plot for principal components 3 and 4 with the addition of a special hidden layer, often called the context
process data. or state layer [Pham2], which acts as a short-term dynamic
memory responsible for the ability to learn to recognise and
generate temporal patterns, as well as spatial patterns [Elm3].
1HXUDOQHWZRUNV This memory, which is simply a copy of the hidden layer, acts
as an extension to the input layer, holding the feedback
The foundation of building intelligent sensors is that some signals from the hidden units from the previous time step. The
variables do not have on-line sensors or the available sensors outputs of these context neurons are then fed forward to the
are nor cost-effective or reliable due to operating conditions. hidden layer for the current time step. Since only the forward
However, they often have a functional relationship with other connections are to be adapted and the feedback connections
process variables which can be measured on-line. It is, are preset to constant values, then these networks can be seen
therefore, clear that large benefits could be obtained by as feedforward networks and the above BP algorithm can be
applying neural networks to data prediction. used [Pham2].
Two learning algorithms have been used for neural
< network approximation of a non-linear activation function f.
; 3URFHVV
The standard and well known backpropagation algorithm
(BP), and the Levenberg-Marquardt method. The former is
$11 1HXUDO)'' basically a gradient descent method, in which the change in
0RGHO V\VWHP the weight of a connection between two neurons is given by
º
wk +1 = wk − α k g k
)LJ Fault Detection and Diagnosis general proposed
scheme.


where w k is a vector of current weights and biases, g k is the
current gradient, and α k is the learning rate. In order to
improve the convergence speed and possible avoidance of
local minima, an additional term, called momentum, is used
in the generalised delta rule. This way, the gradient is also
influenced by the previous weight change.
The Levenberg-Marquardt algorithm, like the
quasi-Newton methods, approaches second-order training
speed without having to compute the Hessian matrix (second
derivatives) of the performance index at the current values of
the weights and biases. Instead, the Hessian matrix is
approximated as
T
H =J J
and the gradient computed as
g = JTe
where J is the Jacobian matrix, which contains first
derivatives of the network errors with respect to the weights
and biases, and e is a vector of network errors. The Jacobian
matrix can be computed through a standard backpropagation
technique that is much less complex than computing the
Hessian matrix.
Whichever the method chosen, before training a neural
network, data preprocessing is required. The first step is to
identify and remove outliers as there is no guarantee that
network training will ignore outliers and fit well on good
samples. Further, since neural networks are nonlinear
approximators, it is experienced that they are more sensitive
to outliers than linear regression methods because they can
bend to reach the outliers [Qin4]. Table 1 shows the
minimum, maximum and the average values for each one of
the process variables. It can be easily seen that each variable
covers a different range of values. Therefore, the errors due to
the higher valued variables will have a greater effect during
training than the errors due to the lower ones, as their
magnitude will be greater. If we are able to ensure that each
variable covers the same range than we also ensure that the
errors on each variable contribute with the same proportion to
the change in the network weights.
0LQLPXP 0D[LPXP $YHUDJH
YDOXH YDOXH YDOXH
The purpose was to train a network that could predict the
process behaviour in terms of the injection time and cushion
parameters, as these were found to be related to the detected
faults. All data points were previously scaled to zero mean
and variance one.
Obtained Epochs
SSE
(OPDQ 0.6054 500
7'11 0.2544 500
7DEOH – Network convergence evaluation after 500 epochs.
To select the best injection moulding fitting model,
several neural networks were built and trained on 781 data
samples using the BP algorithm. TDNN and Elman network
architectures were used and tested on a validation set
constituted by 784 independent samples. Although both
network types have similar performances, the TDNN, with 4
input units, 15 hidden units, with tan-sigmoid activation
functions, and 2 output units, with linear activation functions,
exhibits lower training times, converging more rapidly to the
desired previously set error value. Table 2 shows the
networks performance in terms of sum-squared error (SSE)
after 500 epochs. Figures 7, 8 and 9 show the obtained results
for the TDNN. Particular attention should be given to figure
8, where it can be seen that the network outputs are quite
close to the real data except for a number of points close to
sample 100, which can be explained by a slight deviation of
the measures (see figure 2). The same analysis can be
performed on the error histograms. While for the first output
the network exhibits a healthy behaviour with a peak at
around zero, falling off as the number of errors of greater
magnitude diminishes [Swi1], for the second output that does
not happen, which can be explained by the variability of the
process variable cushion.
&\FOH7LPHV 34.000 245.000 37.975

'RVDJH7LPHV 8.940 9.9000 9.341

,QMHFWLRQWLPHV 6.010 6.790 6.654

&XVKLRQPP 7.400 15.400 14.622 )LJ Error evolution during the training process.


7DEOH – Data measurement ranges.


 0HDVXUHPHQWGDWD [1HWZRUNRXWSXW RRXWSXW $XWRFRUUHODWLRQIXQFWLRQRISUHGLFWLRQHUURURXWSXW
 

 

 

 
              
3UHGLFWLRQHUURU ODJ
+LVWRJUDPRYHUSUHGLFWLRQHUURUV
 

 

 

 
               

0HDVXUHPHQWGDWD [1HWZRUNRXWSXW RRXWSXW $XWRFRUUHODWLRQIXQFWLRQRISUHGLFWLRQHUURURXWSXW

 







 
              
3UHGLFWLRQHUURU ODJ
+LVWRJUDPRYHUSUHGLFWLRQHUURUV
 

 

 

 
               

)LJ Network outputs and error prediction results. )LJ Error correlation plots and histograms.

&RQFOXVLRQV
Linear modelling of the existing real-world problems in
industry is inadequate, since they are non-linear, time-varying
and uncertain.
This paper addresses a system architecture which
combines PCA with the so-called intelligent system
methodologies such as neural networks.
Neural networks can account for temporal dependencies
directly via a delay window or indirectly using a recurrent
copy of the hidden layer. For time series in which temporal
dependency is large, it can be appropriate to use recurrent
networks with time delayed inputs.
The data transformation by PCA accounts for obtaining
more precise and reasonable numerical results. Further
analysis for diagnosis requires the development of a neural
network system able to respond in real-time to the evolving
industrial environment.
$FNQRZOHGJHPHQWV
The authors would like to thank PLASFIL - Plásticos da
Figueira, Lda. for the availability of the mould and for
making the moulding data available. The authors would also
like to express their gratitude to PRAXIS XXI, Sub-Programa
Ciência e Tecnologia do 2º Quadro Comunitário de Apoio,
which financially supports this project.
5HIHUHQFHV
[Swi1] Kevin Swingler, “Applying Neural Networks: A
practical guide”, Academic Press, (1996).
[Pham2] D. T. Pham, “An introduction to artificial neural
networks”. In A. B. Bulsari, Neural Networks for Chemical
Engineers, Elsevier, (1995).
[Elm3] J. L. Elman, “Finding structure in time”, Cognitive
Science, , pp 179-211, (1990).
[Qin4] S. Joe Qin, “Neural Networks for Intelligent Sensors
and Control – Practical Issues and Some Solutions”, In Omid
Omidvar and David L. Elliot, Neural Systems for Control,
Academic Press, (1997).