Automated Regression Test Generation Using
Record & Replay Approach: A Case Study on
Train Control Management System
Adnan Čaušević Rikard Land
Mälardalen University Maximatecc AB
Västerås, Sweden Västerås, Sweden
adnan.causevic@mdh.se rikard.land@maximatecc.com
Abstract—Organizations tend to limit their investment in test
automation due to the lack of information on the actual tests
reuse and when will automated tests really pay off. However,
to perform efficient regression testing of software systems, it is
expected from a development team to posses a certain level of
test automation infrastructure in place, where at the minimum
the test execution is scripted and automated.
In this paper we are proposing the usage of record & replay
approach to observe the functional usage of a component under
test, while its being invoked as part of the whole system or
only in a certain portion of it. Afterwords, executable tests
are automatically derived, containing both test inputs and test
verdict, allowing its later usage as part of a regression testing.
With as minimal effort as one manual test execution, developers
are provided with automated tests, minimizing any concerns on
the investment in automation. A case study from Bombardier
Transportation is provided showing how the proposed approach
substantially reduced the test effort needed when performing
regression testing of the train control management system for
the Stockholm C30 metro train.
I. I NTRODUCTION
With every new release of a software system, a dedicated
testing activity should be performed in order to verify that the
changes to the code, due to fault fixes or introduction of new
features, did not corrupted any of the existing functionality.
Often, we refer to this activity as a regression testing. Due
to its repetitive nature, by over and over re-verifying that the
same functionality still has a correct behavior, efficiency of
regression testing is heavily reliant on the level of automation
a certain organization posses. Depending on the application
domain and the maturity level of a specific organization, this
can vary substantially.
Test automation in industry is often perceived as a mean to
perform automated execution of the system under test (SUT)
by unattended running of a set of test scripts. An execution of
a single test script usually involves retaining SUT in a certain
state by invoking it with a predefined set of input values. A
verdict or a decision whether a test has passed or not, also
known as a test oracle, could be automated as well. However,
this might require an additional investment in automation
depending on the testability of the SUT. Testability defines
how observable the states of the SUT are when performing
Ola Sellin
Bombardier Transportation
Västerås, Sweden
ola.sellin@se.transport.bombardier.com
testing and, for some organizations, manual inspection of the
log files could be the only way to make a verdict on a test.
The goal of test automation is to advance in automating
all of its elements: creation of test scripts, test execution,
generation of test inputs and defining of a test oracle. By not
having automated some of the mentioned elements, the cost
of performing regression testing could substantially increase as
it would require an additional manual effort with every new
software release. On the other hand, it is rather challenging
for companies to change how things are currently done and
especially make a significant jump when it comes to its
own test automation capabilities. Large-scale organizations are
just not eager to change things around easily particularly if
it introduces an additional effort for their employees. This
is why any non-intrusive approach is very welcomed for
consideration.
In this paper we are proposing the usage of record & replay
approach in observing the functional usage of a component
under test (CUT) while the same is executed as part of the
whole system or only in a certain portion of it. Information
sampled during the recording phase are sufficient to automat-
ically generate test scripts, containing both test inputs and
test oracle, and perform their automated unattended execution.
Approach had a positive impact and was seen as an enabler
of a more efficient regression testing of the Train Control
Management System (TCMS) for the Stockholm C30 metro
train developed by Bombardier Transportation site in Västerås,
Sweden.
Organization of the paper is as follows. Section II pro-
vides the background information regarding the specifics of
the TCMS development environment including its simulation
technology. Section III discuss the related work followed by
section IV which provides details regarding the record &
replay approach for automated test generation. Component test
process is outlined in section V while the case study design,
execution and results are presented in section VI. Section VII
discuss possible future directions followed by section VIII
which highlights conclusions on the work presented in this
paper.
Fig. 1. Bombardier’s Mitrac - Train Control Management System (TCMS)1 Fig. 2. An example of an FBD (IEC 61131-3) program structure

II. BACKGROUND of the distributed system that controls the train. Examples of
functions controlled by TCMS include collecting line voltage,
The general perception of software intended usage has
controlling the train engines, opening and closing the train
changed over the last decades. Often, software purpose is
doors, and upload of diagnostic data. TCSM is developed
perceived as a medium to help us in creating spreadsheets,
by following the IEC 61131-3 standard [1] for programming
editing graphical content or just entertain us in the form of
of PLC controllers. In particular, Function Block Diagrams
gaming or presenting multimedia. However, people may not
(FBD) are used to implement most of the functionality of
be fully aware to what extent software has a critical role in our
TCMS while Structured Text (ST) programming language is
lives. Today, software is used to control brakes of a car, the
used for the development of user specific libraries.
speed of a train or the altitude of an airplane. The main reason
FBD is a graphical programming language fitted very well
for this change in software usage has been the increased need
for developers having background in control engineering.
for more complex functionality in products usually perceived
Behavior of a program in FBD is described by connecting
as a physical, or hardware-oriented. Nowadays companies are
input signals to the network of functions and function blocks
using general purpose or custom made hardware modules
which computes the resulting outputs. An example of an
from suppliers on top of which their own in-house developed
FBD program structure is shown in figure 2. It is important
software solution (usually referred as an embedded software)
to distinguish the main difference in the behavior between
often controls the main functionality of the final product.
functions (for example AND, OR elements in fig. 2) and
Programmable logic controllers (PLC) are examples of such
function blocks (for example TP, TON, RS elements in fig.
systems.
2) in an FBD program. Functions always compute the same
A typical PLC system consists of a processor, a memory,
output for a given input vector. Function blocks however have
and a communication bus. Its main advantage is a build-in
internal states causing output to differ even if the input vector
support for input and output (I/O) communication. Having
is the same. This is because the outputs of function blocks
sensor values as inputs and actuator systems as outputs, PLCs
are dependent on the ordering within the sequence of input
are ideal for monitoring and supervising control systems. A
vectors provided to it but also dependent on when (in time)
program running on a PLC is almost always executed in a
a specific input vector is provided to the system and for how
cyclic loop with three distinguish phases: (i) reading all inputs
long (duration) it had that value.
from an input vector, (ii) executing program computation
This behavior of FBD programs emphasize the need to
without any interrupts, and (iii) writing updated values to the
have a structured approach when performing its testing. In
output vector.
particular, re-testing requires a well defined test steps clearly
One example of a PLC product is the Bombardier Trans- indicating sequence of values and timing information to suc-
portation’s Mitrac1 Train Control and Management System cessfully observe proper functioning of the underlying com-
(TCMS) depicted in figure 1. TCMS is a high capacity, infras- ponent under test (CUT). A challenging task, considering that
tructure backbone built upon an open standard IP-technology FBD components are part of a bigger system that is running
that allows an easy integration of all control and communica- on a dedicated PLC hardware. To avoid these problems,
tion functions on-board the train. It is considered as the center simulation techniques are extensively used in software testing
1 http://www.bombardier.com/en/transportation/products-services/ of embedded products [2]. Simulation enables execution of
propulsion-controls/products/train-control-and-management-system.html embedded software on a general purpose PC, enabling utiliza-
 Fig. 3. Maximatecc Simtecc Simulation Technology
tion of existing debugging and test automation solutions. In
addition, it also helps improving testability of these systems
by exposing internal input and output signals.
In the following subsections a short explanation is presented
on how a simulation environment was created in Bombardier
Transportation as part of their SoftTCMS framework by uti-
lizing SimTecc simulation technology by Maximatecc.
A. Maximatecc’s SimTecc
The technical concept of SimTecc simulation environment,
formerly known as CCSimTech [3], is based on the software
implementation of device-drivers. Examples of such are read-
ing and writing of inputs and outputs, accessing the internal
memory of the device and communication via controller area
network (CAN) bus or local area network (LAN). The em-
bedded software is developed to use a well defined interface,
the so-called Hardware Abstraction Layer (HAL) which itself
delivers all the data to and from the hardware. The figure
3 visualizes this concept. The drivers are exchanged in soft
environment, and the new drivers don’t access real hardware
but instead communicate with a virtual software layer. The
embedded software (application) is exactly the same as on the
target system, using the exactly same interface. However, the
soft drivers now enables running of the entire software on a
standard PC. These virtual drivers are the core of maximatecc’s
simulation environment SimTecc. In particular, for the scope of
this paper, we will focus on one specific driver called SimIO.
SimIO driver serves to simulate hardware IO often used
by embedded applications to communicate with peripheral
units, such as motors, controls, switches etc. Maximatecc’s
implementation of simulated IO uses shared memory created
by a dynamic load library (DLL) file. This memory contains
information about the IO signal, such as name and its value and
any other node in the system (who instantiated the same DLL
file) can access this value. This way it is possible to create a
new IO signal (e.g. analogue, digital, pulse-width modulation
(PWM), or any other signal with the pulse nature) that can be
read or set by other processes.
B. Bombardier’s SoftTCMS
Bombardier Transportation site in Västerås, Sweden, deliv-
ers software for Vehicle Control Units (VCU) and Intelligent
Display Units (IDU) to be used on-board the train. The
Fig. 4. General Architecture of the TCMS [4]
intended purpose of this software, once it is loaded into VCUs,
is to communicate with various train subsystems, among
others: brakes, doors, air control units, lights, etc. A general
architecture of TCMS is depicted in figure 4. Palmieri et al. [4]
presented in their work more high-level description of various
TCMS elements and its intended usage.
As previously mentioned, TCMS is considered as a Pro-
grammable Logic Controller (PLC) unit and as such it is
developed by following the IEC 61131-3 standard, in specific
FBD language. In order to simulate VCUs and execute the
corresponding TCMS software loaded on top of them, Bom-
bardier invested in creating several applications for the purpose
of replicating VCUs behavior on a general purpose PC. This
allowed execution of the same TCMS software in both real and
simulated environment. The simulated counterpart was named
SoftTCMS.
MITRAC CC tool is used for the development but also
for the compilation of the code for the real platform. This
is where processes for the real and the simulated compi-
lation diverge. The simulated counterpart modifies the code
to support its compilation for the PC platform (in specific
Microsoft Windows environment) but also adds various glue
code to it. Among others, it enables utilization of the SimTecc
simulation environment, in specific SimIO driver. This, as a
result, increase significantly testability of the TCMS in the
simulated environment and enables efficient debugging and
testing processes.
For the debugging activities, Bombardier previously devel-
oped a tool called DCUTerm. What is interesting with this
tool is that it was created prior to the SoftTCMS environment
and its main purpose was to communicate and debug the
real hardware VCUs. Now, the same tool is used for the
communication with the simulated environment without any
modifications to it.
In order to perform unit, integration and system level
testing of the TCMS software, a dedicated tool was created,
called Mitrac Desktop Component Tester - MDCT. Its intended
usage is to visualize FBD execution flow and allow manual
interaction with the TCMS on a signal level.
 III. R ELATED W ORK
IEEE defines regression testing as: selective retesting of a
system or component to verify that modifications have not
caused unintended effects and that the system or component
still complies with its specified requirements [5]. Manual
regression testing on a subsystem level is a very tedious and
time consuming task given a situation where requirements
are specified from a user or a system level perspective [6].
A record & replay approach to software testing is a well
known technique that has been researched and is in use. The
most common usage scenario for this technique in software
testing is for performing automated graphical user interface
(GUI) testing [7]. Also, reproducing software execution for
debugging purposes is another approach where record and
replay has been already investigated [8].
In [9] Bauer et al. demonstrate an approach that supports a
complete chain from a requirements document to a statistical
test report with a very high degree of automation. The initial
part requires some human involvement in order to convert
requirements into a precise specification, and develop the
PROVEtech TA test runner files. They were able to provide
fully automated test case generation, test execution, and test
evaluation based on a system usage model. The model has
been applied on a real mirror control unit of a car door for
reliability estimations, where messages have been broadcasted
via CAN bus. The presented work describes a practical way of
using rigorous methods to achieve a high degree of automa-
tion.
In order to provide a better understanding and easier de-
bugging of the multi-threaded distributed Java applications,
Konuru et al. have developed a replay of the recorded ap-
plication execution using logical thread schedules and logical
intervals [10]. The described approach is based on the already
existing system called DejaVu that provides deterministic
replay of multi-threaded Java programs on a single Java Virtual
Machine(JVM). However, the framework described in [10] is
developed to support distributed Java applications running on
multiple JVMs. The approach is intended to be independent
of the underlying operating system and not to require any
modifications from the user application in order to enable
replay.
In [11] Netzer et al. present an approach that enables tracing
and replaying the order in which messages in a program are
executed, while debugging it. In this way the program is forced
to reproduce messages in their original order. This approach
focuses only on the messages that are part of the trace to be
analyzed by using a built-in run-time decision mechanism. The
approach introduces a certain level of adaptivity, by continuous
decision making about what to trace. The approach reduces
trace size and enables debugging with substantially lower
execution time overhead.
Ronsse et al. [12] describe a system called RecPlay that en-
ables record/replay of shared memory applications containing
no data races (a program error occurred due to the existence
of synchronization operations). They record the order of all
ƉƉůŝĐĂƚŝŽŶůĂLJĞƌ ƉƉůŝĐĂƚŝŽŶůĂLJĞƌ ƉƉůŝĐĂƚŝŽŶůĂLJĞƌ
ZĞƉůĂLJ ^ƵďƐLJƐƚĞŵ
^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ
^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ
ZĞĐŽƌĚĞƌ
^ƵďƐLJƐƚĞŵ
^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ ^ƵďƐLJƐƚĞŵ
,ĂƌĚǁĂƌĞůĂLJĞƌ ,ĂƌĚǁĂƌĞůĂLJĞƌ ,ĂƌĚǁĂƌĞůĂLJĞƌ
Fig. 5. Overview of the record & replay approach
synchronization operations using Lamport timestamps [13].
During the replay phase the file with timestamps is consulted
in order to track synchronization operations. In this way the
execution of a synchronization operation is delayed until all
synchronization operations with a smaller timestamp have
been executed. This mechanism makes sure that all synchro-
nization operations are executed in an order that guarantees a
faithful re-execution if no data races are present.
The intended purpose for most of the previously presented
contributions in academia is either to perform regression
testing on a GUI level by recording mouse movements and
keyboard logging, or to replicate the exact behavior of the
system in order to localize the fault, a process known as
debugging. Although, the principle of recording and replaying
is still the same, the main difference with our approach is
that we are expecting the system to function correctly, for a
given set of requirements, when the recording is performed.
Afterwords, when replicating the system execution we are not
doing it for the purpose of finding faults, but instead to confirm
that the system is still functioning correctly for the same set
of requirements on a new system release.
IV. R ECORD & R EPLAY
In this section we are presenting a high level overview of
the record & replay approach used for automated generation
of regression tests as depicted in figure 5. Overall idea is
that when a specific subsystem or a component is being
functionally tested, either in isolation or when used by a
complete system, we would like to record interactions with
that component. This way, once we have a new version of
the component, we can replay those interaction in isolation
and in an automated way and thus validate if any changes in-
troduced to the component are corrupting previously working
functionality. In order to further elaborate this approach, three
subsections are following, each discussing distinct phases of
the approach, namely: (i) recording phase, (ii) replaying phase,
and (iii) test verdict (oracle) phase.
A. Recording
In order for the component under test (CUT) to be correctly
observed during its execution all input signals, that could
provoke change in the behavior of the CUT, and all output
signals, that could be changed as a result of the behavior of
the CUT, must be monitored. In most cases this referees to the
signals directly connected to the component itself. However,
there are cases where CUT internal state could be changed
by provoking another component via a specific signal and

R ECORDING STRUCTURE
Unique Signal Sampled Inputs
Value 1 {{In 1,Value}, {In 2,Value}, ..., {In M,Value}}
Value 2 {{In 1,Value}, {In 2,Value}, ..., {In M,Value}}
Value N {{In 1,Value}, {In 2,Value}, ..., {In M,Value}}
making an indirect change through a shared memory or a
similar concept. In such cases, those specific signals must be
monitored as well.
Monitoring, or sampling, of the selected signals has to
be done relative to the global time-related or cycle-related
unique signal. This could be, for example, a global clock or
a cycle counter in the system. Precision of the sampling is of
course very important but too precise recording may result in a
overflow of unnecessary information making it rather difficult
to efficiently generate tests. If we have a system designed to
be executed at the millisecond precision, there is no need to
sample our recording at the microseconds level.
Another important aspect of the recording phase, which also
contributes to a more efficient test generation, is performing
selective data sampling. This means we do not save all values
of monitored signals throughout their execution lifespan, but
only values when there was a change in the signals being
monitored at a specific time. The change could be that one or
more signals at the input of the CUT have a new value, and/or
one or more signals at the output of the CUT has a new value.
Immediately when this is detected a new row in the recording
structure needs to be populated.
The structure of the recording is presented in Table I.
In addition to the sampled inputs, outputs and the unique
signal values, each row contain information called Oracle and
Invocation. Oracle represent a Boolean flag indicating if a
specific row in the recording structure should be used for a
verdict and it is set whenever there is a change on any of the
monitored outputs. Invocation also represent a Boolean flag,
but it is indicating if a specific row in the recording structure
should be used for provoking the CUT during the replaying
phase and it is set whenever there is a change on any of the
monitored inputs.
B. Replaying
Replaying phase, as its name suggest, serves for the purpose
of exercising the CUT in the same manner as it was performed
during the recording phase. When performing replaying, only
a subset of sampled recording structure is used. Essentially,
every recorded sample that has a flag Invocation set to True is
used for replaying. Idea is to provoke the CUT by setting the
signals to specific values at the specific point in time. For that
to achieve it is important to keep track of the relative offset
for the unique signal used as a timing reference. For example,
during the recording phase, the first row in the recording
structure could have been observed when the unique signal had
TABLE I
Sampled Outputs Oracle Invocation
{{Out 1,Value}, {Out 2,Value}, ..., {Out K,Value}} {True/False} {True/False}
{{Out 1,Value}, {Out 2,Value}, ..., {Out K,Value}} {True/False} {True/False}
.
.
.
{{Out 1,Value}, {Out 2,Value}, ..., {Out K,Value}} {True/False} {True/False}
a value of 3000. This value can represent number of cycles
passed from the start of the execution. When replaying, we do
not have to wait for the system to reach 3000 cycles in order to
start invoking our CUT. We can provoke the CUT immediately
after it is initialized, but every subsequent invocation of the
CUT has to happen relative to that number.
What is specific about replaying phase is that at the same
time as we are provoking the CUT, by automated replaying
of previously recorded signal values, we are also performing
a new recording of the CUT in parallel. This new recording
is done exactly the same as in the first phase since recording
itself cannot distinguish if the CUT is provoked by a user, by
another component or by a previously recorded sample. By
doing so we can create a new structure of recorded samples
allowing us to perform its analysis and comparison with the
original recording and thus enable creation of automated test
oracle.
C. Test Oracle
Once we have recorded structures from both the recording
phase and the replaying phase, we can devise a test verdict.
This is done by comparing these two structures as described
in the next steps:
1) Both structures are filtered out such that only rows which
have an Oracle flag set to True are left for the analysis
(as per table I).
2) Values for the unique signals are compared having in
mind that relative offset is important rather than the
actual value.
3) Values for each and every input and output signal are
compared between the structures.
4) If any of the above comparison is not true, the test is
reported as failed. Otherwise, test is passing.
For systems having timing and state-full nature it is very
important to preserve the order of input signals which are
invoking a CUT. In addition to the exact sequence of input
vectors, it is also important to maintain the same timing offset
from a single reference point such that input signals are not
only provided in a specific order but also at the specific
time. This, as a result, makes it rather difficult to create a
flexible test oracle. Essentially, any time-related deviations
when observing outputs may result in a false positive test
case: a failing test case which is actually not reveling any
faults in the system, the signalwas just set late. For that reason
it is important to consider introducing tolerance factor when
performing automated test oracle generation. This factor could

Fig. 6. Mitrac Desktop Component Tester - MDCT tool visualizing FBD
execution flow and allowing developers to manually override a signal value
when performing functional testing.
be based on the unique signal values such that there is a
tolerance in when a specific values for outputs are observed.
For example, if we are having microseconds precision of
the unique signal, we could state that our system tolerate
deviations of ±10 microseconds when observing changes to
the output signals.
V. C OMPONENT T ESTING P ROCESS
Before presenting the case study design, how it is executed
and the results we collected in the following section, we need
to first elaborate on how the current component testing process
is performed in Bombardier Transportation.
TCMS is considered as a safety-critical product used in
the railway domain and thus its development, testing and
safety assessing obligations are regulated by the EN50128
standard [14]. Among other requirements, it is mandated by
EN50128 to perform functional testing of the software on the
lowest (unit) level. In the case of Bombardier, a unit is a
function block diagram (FBD) implementation in IEC 61131-
3 standard which is not composed of other FBDs (unless
they belong to the library of already tested, reusable FBDs).
However, what is important to point out here is that unit
level testing, or component level testing as its referred to in
Bombardier internally, is performed by developers itself. This
by no means should be a surprise considering that in almost
all programming environments unit level testing is in general
a responsibility of a developer. However, the lack of unit
testing framework for function block diagrams makes it rather
difficult, if not impossible, to have any sort of automation
support. This is why currently the component testing process
is devised as following:
1) Based on the list of provided requirements, developer
creates a Software Component Test Specification docu-
ment.
2) Based on the same list of provided requirements, an
implementation in an FBD language is conducted by
a developer.
3) Functional testing of a component is performed manu-
ally by following the Software Component Test Speci-
fication document and using MDCT tool (presented in
figure 6).
4) Results of functional testing are documented as an Excel
sheet and also in a new document called Software
Component Test Record.
The main reason why so much attention is given to various
documents throughout the development of a component is the
safety auditing process. This process is mandated for safety-
critical products like TCMS. However, main problem is due
to the fact that these documents cannot be used or re-used
for any other purpose. This is very much emphasized when a
component needs to be updated, due to the change request. In
such a situation, the following process for the development is
in place:
1) Based on the change request, and update is done to the
Software Component Test Specification document.
2) Actual change is implemented in the FBD code.
3) Complete functional testing of a component is per-
formed manually by following the Software Component
Test Specification document and using MDCT tool.
4) New results of functional testing are documented as an
Excel sheet and also in a new Software Component Test
Record document.
This means that any single update to any existing component
requires a full re-testing activity which at the moment could
be done only by a manual effort. However, just because an
additional manual effort is spend on each update, this does
not imply that it has to be immediately automated. Investing
in automation is very much dictated by the number of re-runs
that could potentially occur. For this purpose we investigated
the ratio of component updates for the existing C30 project,
currently being developed, and compared it with the finalized
VZI300 Zefiro (high speed train) project. Results are presented
in table II.
TABLE II
O CCURRENCE OF UPDATES ON A COMPONENT LEVEL
Updates C30 Project (in %) VZI300 Zefiro Project (in %)
0 57,46 65,71
1 28,95 0,00
2 4,98 18,22
3 5,88 8,87
≥4 2,63 7,2
At first it may seem that any effort in automation is not
really motivated by the figures provided. However, Bombardier
Transportation is always trying to reuse components from the
previous projects, leading to a situation that quite a number of
components do not have any change requests even all the way
up to the usage on the real train. But those that do have updates
are the ones specific to a train project and in most cases they
do have a higher complexity (in terms of number of inputs,
outputs, function blocks, etc.). Higher complexity naturally
leads to a higher effort when performing manual testing.

VI. C ASE S TUDY

In order to quantify potential benefits that could be gained
by using the record & replay approach for automated test
generation, a case study was performed on the train control
and management system (TCMS) product for the Stockholm
C30 metro train developed by Bombardier Transportation site
in Västerås, Sweden.

A. Case Study Design

The goal of this case study is to evaluate applicability of
the record & replay approach on a real industrial system that
had actual change requests, which required from developers
to conduct regression testing of the same. Focus is on: (i) es-
timating possible saving effort when performing regression
testing, and (ii) evaluating correctness of the proof-of-concept
implementation. In order to achieve such goals, case study was
designed by following the next steps:
1) Implementation of the proof-of-concept tool supporting
the record & replay approach for a given industrial
system.
2) Selection of the component under test (test object).
3) Recording of the tests for the initial release of the CUT,
based on the Software Component Test Specification.
4) Replaying recorded tests on every subsequent release of
the CUT.
5) In case some tests fail at the specific release, a new
recording is done for that specific test based on the
Software Component Test Specification for that release.
Details on each of the above steps are presented in the
following subsections.
B. TestRecorder implementation
To practically support monitoring, recording and replaying
of signals in the TCMS environment, a dedicated tool was cre-
ated. TestRecorder interface is depicted in figure 7. Inputs and
outputs areas are drag-and-drop enabled allowing developers
to easily decide which signals to select from MDCT tool for
monitoring.
Open Test button allows opening of previously saved tests
to be used for replaying or even new recording in case the
same signals could be used again. Record and Replay buttons
are implemented according to the description of the record
& replay approach in section IV. Save as Test button allows
saving of the recorded test for later use. Below the buttons,
an information panel display results of test replay execution
in terms of a pass or a fail message.
C. Test Object Selection
Selection of the test object for the case study is based on
the following criteria:
• number of change requests it had,
Fig. 7. TestRecorder tool - A proof-of-concept implementation for the record
& replay approach used within the case study.
• number of input and output signals it is designed to
handle, and
• number and diversity of function blocks it contains.
The component selected for the case study is part of
the Traction Brake sub-level function group. It is named
DBC LV MchSup and its purpose is:
To supervise master controller handle of the train and calcu-
lates the traction/brake reference from the master controller
handle unit. Additionally, it prioritizes the inputs from master
controller unit and validates the status of a master controller.
Finally, it generates events for deviations of micro switches
from master controller and master controller handle cutouts.
When it comes to the change requests, this component had
in total 6 versions2 internally released for testing. The first
release was introduced 12th of September, 2014 and the last
modification was done on 23rd of August, 2015.
Complexity of the selected CUT, expressed in the terms of
number of signals and function blocks is listed here:
• 16 input signals - 14 Boolean and 2 integer types
• 9 output signals - 6 Boolean and 3 integer types
• 14 parameters (constants) - 1 Boolean, 1 time and 12
integer types
• 14 function blocks (state-full elements in FBD)
• 38 functions (state-less elements in FBD)
By discussing internally with developers at Bombardier to
what extent this component would be a good representative,
it is stated that this component is considered of a higher
complexity than what an average FBD diagram looks like.
2 This was at the moment of publication submission for a review process.
Development on the C30 TCMS project was not completed at that time.

Fig. 8. Trace logging result when manually recorded Test 7 is replayed.
D. Case Study Execution
After selection of the CUT, the case study was executed by
first determining which previous internal releases of SoftTCMS
contained changes to the selected CUT. Each and every release
was individually compiled and verified that it could be used
by Mitrac Desktop Component Tester (MDCT) tool.
The very first initial release in which DBC LV MchSup
CUT was introduced was selected as a starting point for the
recording. Based on the Software Component Test Specifica-
tion, six individual tests (Test 1 to Test 6) were recorded by
setting specific signals to the stated values with the MDCT
tool. Essentially, we were replicating the exact manual effort
spent by developers.
Subsequently, for every next internal release, which con-
tained updates to the selected CUT, recorded tests were
replayed. For these activities, only a machine effort was
used. Figure 8 displays trace logging done by DCUTerm tool
when previously manually recorded test is now automatically
replayed. In case a specific test was failing on a given release,
a new manual recording was done based on the Software
Component Test Specification for that release and for that test.
For every other test which was passing, no additional actions
were needed.
Since the Software Component Test Specification for
the last internal release contained one additional test for
DBC LV MchSup CUT, it had to be manually recorded for the
first time. However, there were no subsequent releases where
this test could be replayed.
Results of the case study execution process are outlined and
discussed in the following subsection.
E. Results
In table III, test execution results are displayed for the
selected CUT. It contained seven tests spanning over six
releases. However, Test 7 was only introduced in the last
release and no subsequent regression testing was done with it.
Every test that was recorded for the first time has a label Initial
Recording. A failing test required new recording and thus such
situations are labeled as Fail / New Recording. Passing test is
labeled as Pass.
Looking at the occasions where regression testing had to
be performed, we have in total 30 regression testing activities
(all items except Initial Recording). Out of those 30 regression
tests, 24 are passing when being automatically replayed using
the proposed record & replay approach and the TestRecorder
tool.
This means that 80% of the regression testing activities could
have been performed completely automatically without any
need for the manual effort to be spend, if the record & replay
approach was used from the very beginning of the component
development.
It is also interesting to discuss the tests which were failing
and the reasons for that. Test 6 was failing in Version 2 due
to the miss-implementation in previous release. This of course
required a new recording to be performed based on the new
test specification. Afterwords, the new recording was passing
on all subsequent releases. The Version 3 release had most
failing tests: Test 1, Test 2 and Test 5. Reason for that is a
change request regarding component interface. Several input
and output signals were named slightly different in the Version
3 release. This is an interesting situation with respect to the
regression testing as the reason for the change is not really
functionality driven. In such a case there is no need to do
a completely new recording as it is easier to just change
the name of signals in the recorded test script file. By re-
running updated test recordings on the Version 3 all tests
where passing. In addition, tests where also passing on all
the subsequent releases.
In the Version 4 release, the selected CUT had a change
request regarding its functionality, where the input range for
master control handle was not any longer between -100 and
100, but instead -10000 and 10000 to support the increased
precision. Interestingly, only two previously recorded tests,
Test 3 and Test 4 were failing because of this change. Once
again, new recording of those tests on the Version 4 release
was sufficient to keep that functionality automatically re-tested
in all the subsequent releases.
F. Limitations
In order to fully understand the meaning of the presented
results it is important to discuss limitations of the study
presented in this paper.
External Validity. The component under test used within
this case study was developed in FBD 61131-3 program-
ming language as part of a train control management system
which limits to some extent its external validity. However, no
specifics of FBD programming language influenced how the
record & replay approach is used within the study. As long
as a component under test is observable in terms of input and
output signals or variables, a general principle of the record
& replay approach will apply.
Internal Validity. One of the criteria for the selection of the
component under test was the number of change requests
it had. Considering that the effect of the record & replay
approach increases with the number of change request the
CUT has, this criteria could be considered as a bias in the
study. But, the main reason for having this criteria in the

T EST RESULTS
Version 1 Version 2
Test 1 Initial Recording Pass Fail / New Recording
Test 2 Initial Recording Pass Fail / New Recording
Test 3 Initial Recording Pass
Test 4 Initial Recording Pass
Test 5 Initial Recording Pass Fail / New Recording
Test 6 Initial Recording Fail / New Recording
Test 7 - - -
first place was to highlight the process when using the record
& replay approach rather than its benefits. In the worst case
scenario the record & replay approach would not save any
manual effort but it would neither introduce any overhead as
it is designed to be a non-intrusive approach.
Construct Validity. When measuring the saved effort while
using the record & replay approach we assume that the
same amount of manual effort was spent when performing
regression testing of the CUT. In practice this may not be true
due to the fact that developers when performing regression
testing are to some extent already familiar with the system
and re-running the tests could be done quicker.
Conclusion Validity. In the presented case study, only one
CUT was used for the evaluation of the record & replay ap-
proach. However, the selected CUT does represent a complex
component with a critical function, determining traction/brake
reference based on the position of the driver’s handle. More
components could have been selected for evaluation of the
record & replay approach, but authors do not see how could
that specifically help to increase confident on the applicability
of the presented approach.
VII. F UTURE W ORK
With the record & replay approach, presented in this paper
and applied within the case study, developers at Bombardier
have gained the possibility to replace the manual effort in
the regression testing phase with the machine effort. Since
PLC systems are time-aware, replaying phase took the same
amount of time as it did during the manual recording phase.
This means that several time delays in setting various signals
are not really a consequence of a system behavior but rather
the result of a manual invocation of the system. To emphasize
this, figure 9 is showing how a full execution of Test 7 could
be replayed almost 5 times faster by just a simple removal of
manual delays introduced when system was invoked.
In addition, parts of a PLC component, or even complete
PLC components, may not have any timing related implemen-
tations. They could be constructed using basic functions (logic
operators, comparison blocks, etc.) also known as stateless
elements. This creates a unique opportunity to “speed up” test
execution since any timing information is again purely because
of a user is manually invoking the system. However, in order
to achieve such an advancement in the machine effort needed
to execute a test, one need information on the dependency
between inputs and outputs of a PLC component. Especially
TABLE III
Version 3 Version 4 Version 5 Version 6
Pass Pass Pass
Pass Pass Pass
Pass Fail / New Recording Pass Pass
Pass Fail / New Recording Pass Pass
Pass Pass Pass
Pass Pass Pass Pass
- - Initial Recording
Fig. 9. Trace logging result when Test 7 is executed without unnecessary
manual delays.
information regarding the usage of state-full elements while
propagating from the input to the output. These information
could be collected by analyzing a source code of an underlying
PLC component and creating a dependency matrix between the
inputs and the outputs of a given component.
Even in the case of PLC components containing timing
information, since the complete system is simulated, there
is an opportunity to increase the speed of simulation. This
can additionally save machine effort and obtain a test result
feedback quicker with respect to the real time. In case of the
SimTecc simulation environment, a TimeSync module support
such a possibility.
One additional opportunity to save the machine effort,
resulting in getting a quicker feedback on the regression test
results, is to introduce parallel execution of tests. Again, based
on the analysis of the dependency matrix between inputs
and outputs, several test recordings could be waived into one
parallel replay of the PLC component execution. This can lead
to a significant reduction of machine execution effort.
The current implementation of the record & replay approach
is done as a stand alone tool for the evaluation purposes.
Bombardier’s intention is to have it integrated within the
Mitrac Desktop Component Tester tool where most of the
above mentioned advancements could be easily integrated and
made available for engineers to use.
Although presented case study and proof-of-concept imple-
mentation does rely on specifics of the Bombardier’s TCMS,
there is a possibility to generalize this approach for usage
in other type of systems and even in other type of domains.
What we plan to investigate further is how other customers of
SimTecc simulation environment could benefit from the record
& replay approach, especially in situations where optimization
has to be done during replaying but underlying source code is
not available to support analysis.
VIII. C ONCLUSION
Test automation is often perceived as an expensive but
inevitable investment for various software industry domains.
Even a simple task of automating test execution in an unat-
tended manner could be considered as a significant testing
infrastructure investment. Additional effort needed for writing
of test scripts and analyzing test execution results makes the
decision on test automation investments even harder to make.
Our solution based on the record & replay approach ad-
dresses this problem by reusing the manual effort performed
during the functional testing of a component. By recording
interactions on the specific component for a certain time
duration, we are able to automatically replicate the same
interactions by replaying the execution flow. This, when done
on a newer version of a component, enables efficient regression
testing process.
The record & replay approach does not represent a novelty
in itself. It has been an established method for increasing
automation for many years now. However, to the best of
our knowledge, there are no currently proposals to utilize
record & replay approach for regression testing on a unit
or component level. Most recording efforts on unit level
are addressing debugging aspect of the development process,
while any recording for the purpose of regression testing is
focusing on a GUI level. The same way as we are recording
mouse movements for GUI testing, we can say that here we
are recording signal movements.
Except for enabling efficient regression testing, the record &
replay approach provides an easy entry for companies into the
area of test automation due to the fact being a non-intrusive
method, something which is highly appreciate in the industrial
context. Even if the method would not help in a specific case,
for example when every regression testing instance would
require a new recording, it will neither introduce any additional
manual effort for its usage.
Once the method does show itself useful, it could provide
benefit in terms of saving manual developers effort. In the case
study presented in this paper estimated saving for regression
testing is 80%. In addition, the recording structure of each test
contains sufficient information to almost completely generate
Software Component Test Record documentation needed by
the safety assessors.
Regression testing and regression tests do represent valu-
able assets during the development of any type of product,
especially safety critical ones. Having mechanisms to re-assure
correct behavior helps increasing the speed of the development
process by providing a safety net around existing functionality.
ACKNOWLEDGMENT
The conducted research was supported by The Knowledge
Foundation (KKS) through the AGENTS project (Automated
Generation of Tests for Simulated Software Systems). The
authors would like to thanks Maximatecc AB and Bombardier
Transportation for successfully supporting a research-industry
co-production.
R EFERENCES
[1] I. E. Commission, “Iec international standard 61131-3,” Programmable
Controllers, 2014.
[2] J. Teich, “Hardware/software codesign: The past, the present, and
predicting the future,” Proceedings of the IEEE, vol. 100, no. Special
Centennial Issue, pp. 1411–1430, 2012.
[3] A. Möller and P. Åberg, “A simulation technology for can-based
systems,” CAN Newsletter, no. 4, 12 2004.
[4] M. Palmieri, A. Cicchetti, and A. Öberg, “Cutting time-to-market by
adopting automated regression testing in a simulated environment,”
in The 26th IFIP International Conference on Testing Software and
Systems, M. G. Merayo and E. M. de Oca, Eds. Springer, 9 2014,
pp. 129–144.
[5] “Ieee standard computer dictionary. a compilation of ieee standard
computer glossaries,” IEEE Std 610, p. 170, 1991.
[6] E. Engström and P. Runeson, “A qualitative survey of regression
testing practices,” in Product-Focused Software Process Improvement,
ser. Lecture Notes in Computer Science, M. Ali Babar, M. Vierimaa,
and M. Oivo, Eds. Springer Berlin Heidelberg, 2010, vol. 6156, pp.
3–16.
[7] A. M. Memon, “Gui testing: Pitfalls and process,” Computer, vol. 35,
no. 8, pp. 87–88, 2002.
[8] M. Ronsse, K. De Bosschere, M. Christiaens, J. C. de Kergommeaux,
and D. Kranzlmüller, “Record/replay for nondeterministic program ex-
ecutions,” Commun. ACM, vol. 46, no. 9, pp. 62–67, 9 2003.
[9] T. Bauer, F. Bohr, D. Landmann, T. Beletski, R. Eschbach, and J. Poore,
“From requirements to statistical testing of embedded systems,” in
Proceedings of the 4th International Workshop on Software Engineering
for Automotive Systems, ser. SEAS ’07. IEEE Computer Society, 2007,
pp. 3–.
[10] R. Konuru, H. Srinivasan, and J.-D. Choi, “Deterministic replay of
distributed java applications,” in Parallel and Distributed Processing
Symposium, 2000. IPDPS 2000. Proceedings. 14th International, 2000,
pp. 219–227.
[11] R. Netzer and B. Miller, “Optimal tracing and replay for debugging
message-passing parallel programs,” The Journal of Supercomputing,
vol. 8, no. 4, pp. 371–388, 1995.
[12] M. Ronsse and K. De Bosschere, “Recplay: A fully integrated practical
record/replay system.” ACM Transactions on Computer Systems, vol. 17,
no. 2, pp. 133–152, 1999.
[13] L. Lamport and R. S. Gains, “Time, clocks, and the ordering of events
in a distributed system.” Communications of the ACM, vol. 21, no. 7,
pp. 558–565, 1978.
[14] CENELEC, “50128: Railway application–communications, signaling
and processing systems–software for railway control and protection
systems,” Standard Report, 2001.