Reproducible Testing for Behavior-Based Robotics
Csaba Kertesz1 , IEEE Member, and Markku Turunen1
Abstract Reproducibility is essential in the testing of robots
because verification techniques are required to handle the
growing software complexity. This short paper introduces the
deterministic test cases (DTCs) for extending the standard
testing practices to test incremental changes in behavior-based
systems. Recorded sensor data for DTCs and design patterns
guarantee that the behavior execution is equivalent on the robot
and when the test case is simulated on a computer. These DTCs
can be created easily, their execution is fast, they can test any
kind of robot skills and they minimize the manual testing with
robots in a project. A real world example with a Sony ERS-7
proves that the DTCs can be used successfully in a long run.
I. INTRODUCTION
Traditionally, the robot control software is developed in
simulation environments to spare the operating time of the
expensive robots. In [1], Pioneer 3-DX and PeopleBot robots
were modeled in ROS and Gazebo to run autonomous
navigation task with 3D mapping. These robots had wheels,
therefore, the dynamics modeling was easier than the chal-
lenging legged robots. They found that a proper robot model
in the simulation provided enough precision to deploy the de-
veloped code to the robots. However, the difference between
the simulation and reality makes often impossible to reuse
the same software in both the simulation environment and
the real robot. Laue and Hebbel [2] tried to overcome this
problem by combining the Webots simulation and recorded
data of the robot for optimizing gait parameters, but they
could not reach exact match between the two environments
with genetic algorithm. Araiza-Illan et al [3] used model-
based test generation in ROS and Gazebo to reach high code
coverage during testing and cover rare cases. A limitation of
their work was the static simulation environment and on-line
test generation would be required to cover those cases. As we
can see all kind of solutions are available to develop robot
software, but to the best knowledge of the authors, there
has been a little interest in the literature for reproducible
testing of mobile robots. This paper presents a concept how
the testing process can be enhanced with recorded test cases
of the robot to deal with the increasing complexity of the
behaviors.
II. D ETERMINISTIC T EST C ASES
The proposed deterministic test cases (DTCs) in this paper
does not replace the traditional testing practices. The robot
software still requires unit, module and functional tests, but
1 Csaba Kert
esz and Markku Turunen are with University of Tampere,
33710 Tampere, Finland. (emails: csaba.kertesz@ieee.org and
markku.turunen@uta.fi)
Reproducible Research in Robotics: Current Status and Road Ahead Workshop
ICRA 2017, Singapore
the DTCs complement those with behavior-based integration
testing.
A DTC requires guarantee from the execution model of the
robot software that the algorithms provide the same results
on the robot and on a remote computer. Some basic design
patterns should be followed to ensure this constraint:
Heartbeat (HB): A timer signals the main cycles in the
software when periodic tasks must be executed.
Robot state (RS): This data storage contains the latest
sensor data (e.g. accelerometer, sonar) and the higher
level variables (e.g. emotions, locomotion direction).
Sensor data are collected between heartbeats, but they
are not evaluated until the new heartbeat signal arrives.
The higher level variables are extracted from the raw
sensor data and they are stored to the robot state. To use
DTCs on the robots with limited resources, the robot
state should be constructed as light as possible. For
example, if the environmental brightness is computed
from the camera picture, the data structures of this
algorithm should not be stored in the robot state, only
the final brightness level.
Session log (SL): When the robot state is actualized after
a heartbeat, the new version is saved to the session log.
Furthermore, the behaviors update their states according
to the new robot state and they report their state changes
to the session log as well. A session log collects these
information after the robot is switched on until it is
powered down.
Random numbers (RN): The robot software must report
all generated random numbers to the session log which
effect the behaviors.
Since a session log contains all information (sensor data,
random numbers) what influence the behavior state changes,
the whole session can be replayed back deterministically with
the log on a remote machine.
A. Test Case Creation
A DTC is prescribed with a test case specification, which
Algorithm 1 Deterministic test case creation.
1 . S t a r t t h e r o b o t s o f t w a r e and c r e a t e a h e a r t b e a t t i m e r .
2. t = 0
3 . C o l l e c t s e n s o r d a t a and w a i t f o r t h e h e a r t b e a t s i g n a l .
4 . U p d a t e t h e a c t u a l r o b o t s t a t e ( RSt ) w i t h t h e s e n s o r d a t a .
5 . U p d a t e t h e b e h a v i o r s , s a v e t h e random numbers t o RNt and
t h e b e h a v i o r s t a t e c h a n g e s t o BSt .
6 . t = t +1
7 . Goto 3 u n t i l t h e t e s t c a s e i s f i n i s h e d .
8 . Save t h e s e s s i o n l o g .
 Algorithm 2 Deterministic test case execution.
1 . Load t h e s e s s i o n l o g f o r a DTC and s t a r t t h e r o b o t
software .
2. Create a heartbeat timer .
3. t = 0
4 . I n c r e a s e t h e h e a r t b e a t t i m e r by HBt .
5 . Load t h e c u r r e n t r o b o t s t a t e ( RSt ) from t h e l o g .
6 . U p d a t e t h e b e h a v i o r s w i t h RSt , u s e t h e random numbers
from RNt and s a v e t h e b e h a v i o r s t a t e c h a n g e s t o BS t .
7 . t = t +1
8 . I f t < T1 t h e n g o t o 4
9 . Compare {BS0 , . . . , BST 1 } t o {BS 0 , . . . , BS T 1 } . I f
t h e y a r e e q u a l t h e DTC was s u c c e s s f u l o t h e r w i s e f a i l e d .
illustrates the purpose, and step-by-step procedures. The
session log of a DTC is defined as follows:
SL = (HBt , RSt , BSt , RNt ), t {0, ..., T 1}
where the session log (SL) stores the heart beat intervals
(HBt ), the robot states (RSt ), the behavior state changes
(BSt ) and the generated random numbers (RNt ) for each
t cycle from the robot startup (t = 0) until the test case is
finished (t = T-1). The DTC creation is detailed in Algorithm
1. When the session log is recorded on the robot, the
heartbeat timer issue a signal based on the system time in e.g.
every 30 msecs to evaluate the gathered sensor data since the
previous heartbeat, the robot state is filled with the new data
and the behaviors are updated according to the actual robot
state. The elapsed time between two heartbeat signals (HBt )
are recorded to the session log because it is not guaranteed
that the operating system on the robot can precisely keep the
fixed heartbeat intervals and some behavior functions may
depend on the elapsed time.
As it can be seen, the DTC definition is general, it does
not depend on specific robot hardware or skill. Any situation
can be tested with a DTC forasmuch the executed behaviors
exhibit the robot capabilities to solve a problem.
To describe an example, an obstacle-avoidance DTC can
be created with the above mentioned principles if the session
log is saved while the robot detect and avoid an object.
The robot starts to move forward after powered on (t =
0), an obstacle is detected by the sonar sensor data after
1 meter, object-avoidance behavior is activated on the robot,
the locomotion path is modified, the robot turns away and
the session is terminated after 30 more seconds (t = T-1).
The session log is finished and transferred from the robot to
a computer as an object-avoidance test case.
B. Test Case Execution
When a new change happens in the robot software, a DTC
can be run by the pseudo-code of Algorithm 2 on a regular
PC. There are key differences in the software execution on
the robot and on the computer in testing mode. When the
heartbeats are simulated on the PC, the heartbeat timer does
not take into account the system clock, but the elapsed time
is increased by HBt for each cycle from the session log.
The test case simulation do not have to wait between the
cycles because the robot state for time t (RSt ) is already
available to update the behaviors and the execution is as
Reproducible Research in Robotics: Current Status and Road Ahead Workshop
ICRA 2017, Singapore
quick as possible. The behavior state changes are saved
to BSt since the equivalent execution must be verified by
comparing them to robot version (BSt ). If the behaviors use
random numbers, they are read from RNt . The session log
processing continues respectively until the DTC is finished
(t = T-1). The DTC is successful if the recorded behavior
state changes are equivalent with the stored changes in the
session log.
III. L IMITATIONS
Deterministic test cases can be recorded easily with the
robot regardless of the tested skills, however, this procedure
(1) requires human supervision. An other important fact is that
the behavior state changes in the session log BS0 , ..., BST 1
must be modified if a smaller revision is introduced in the
tested behaviors (e.g. a behavior name is changed). When the
tested behavior or the underlying sensor data processing was
considerable changed in the software, the session log for the
DTC must be recorded again with the robot. Although these
practices have maintenance costs and DTCs are suitable for
finished behaviors, but running a DTC on a PC spares the
manual testing session with a real robot.
IV. E XAMPLE A PPLICATION
DTCs have been used in the AiBO+ project1 for three
years to build a new artificial intelligence (AI) for the
Sony ERS-7 robot dogs. The new features were added
incrementally to the behavior-based system and the previous
skills were protected by testing their recorded DTCs. The
increasing software complexity was also handled by the
dynamic behavior creation with algorithmic decomposition
[4] how the skills can be separated effectively.
15 complex test cases have been defined for verification,
including locomotion functions, obstacle avoidance, touch
responses and motion control from a remote client. The
DTC execution is quick and an AI instance for each DTC
can be tested in separate threads simultaneously. With these
advantages, the 17.5 minutes-long session logs are executed
in few seconds on an average laptop. Because the PC runs
the same robot software in simulation mode, additional code
quality tools (code coverage, memory error checkers) can be
used in conjunction with DTCs.
V. C ONCLUSION
The deterministic test cases were introduced in this paper
to show a behavior-based testing method which runs the
robot software on a PC identically. This method can be
beneficial to develop behavior-based robots for long-term
research or commercial purposes. The proposed behavior
execution model has further advantages in the debugging
and feature extraction for machine learning, but they are out
of the scope in this paper because of the page limitations.
Further work can include the standardization of the robot
1 http://aiboplus.sf.net
 state and the scalability must be analyzed with behavior-
based systems with very high complexity.
R EFERENCES
[1] K. Takaya, et al. Simulation environment for mobile robots testing
using ROS and Gazebo, in Proc. of 20th Intl. Conf. in System Theory,
Control and Computing (ICSTCC), 2016, pp. 96-101.
[2] T. Laue, and M. Hebbel, Automatic parameter optimization for a
dynamic robot simulation, in Robot Soccer World Cup. Springer
Berlin Heidelberg, 2008, pp 121-132.
[3] D. Araiza-Illan, A. G. Pipe, and K. Eder, Model-based Test Gener-
ation for Robotic Software: Automata versus Belief-Desire-Intention
Agents, arXiv preprint arXiv:1609.08439, 2016.
[4] C. Kertesz, Dynamic behavior network, in Proc. of 10th Intl. Symp.
of Applied Machine Intelligence and Informatics (IEEE-SAMI), 2012,
pp. 207-212.

Reproducible Research in Robotics: Current Status and Road Ahead Workshop

ICRA 2017, Singapore

The author has requested enhancement of the downloaded file. All in-text references underlined in blue are linked to publications on ResearchGate.