Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com
ScienceDirect
Available ScienceDirect
Availableonline
onlineatatwww.sciencedirect.com
www.sciencedirect.com
Procedia Manufacturing 00 (2019) 000–000

ScienceDirect
ScienceDirect 
Procedia Manufacturing 00 (2019) 000–000 www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia
Procedia Manufacturing 28 (2019) 121–126
Procedia Manufacturing 00 (2017) 000–000
www.elsevier.com/locate/procedia
International Conference on Changeable, Agile, Reconfigurable and Virtual Production
International Conference on Changeable, Agile, Reconfigurable and Virtual Production
Digital twin for adaptation of robots’ behavior in flexible robotic
Digital twin for adaptation of robots’ behavior in flexible robotic
assembly
Manufacturing Engineering Society assembly lines 2017, MESIC 2017, 28-30 June
International Conference
lines
2017, Vigo (Pontevedra), Spain
Niki Kousia, Christos Gkournelosa, Sotiris Aivaliotisa, Christos Giannoulisa, George
Niki Kousia, Christos Gkournelos Michalos
a
, Sotiris
a
, Aivaliotis
Sotiris Makris , *Christos Giannoulisa, George
a a,
Costing models for capacity
a Michalos optimization
a
, Sotiris Makrisin a, Industry 4.0: Trade-off
*
Laboratory for Manufacturing Systems & Automation, Department of Mechanical Engineering and Aeronautics, University of Patras , Patras
a between used capacity and operational efficiency
26504,
Laboratory for Manufacturing Systems & Automation, Department Greece
of Mechanical Engineering and Aeronautics, University of Patras , Patras
26504, Greece

Abstract A. Santana , P. Afonso , A. Zanin , R. Wernke

a a,* b b

Abstract a
University of Minho, 4800-058 Guimarães, Portugal
Product personalization is the global trend driving b the development
Unochapecó, of modern
89809-000 Chapecó, production systems. This in turn creates the need
SC, Brazil
to deploypersonalization
Product flexible and reconfigurable assembly
is the global trend drivingsystems. New type of factories
the development exploiting systems.
modern production the capabilities
This inof multiple
turn createsresources
the need
such as human
to deploy operators
flexible and mobile and
and reconfigurable / or stationary
assembly systems. robot
New assistants are emerging.
type of factories Thisthe
exploiting paper investigates
capabilities the use resources
of multiple of digital
world
such asmodelling techniques
human operators andinmobile
such hybrid
and / or production
stationarysystems for enabling
robot assistants are system reconfiguration
emerging. through shared
This paper investigates environment
the use of digital
and process
world
Abstract modellingperception.
techniques The suggested
in such hybriddigital worldsystems
production model for infrastructure involves
enabling system three mainthrough
reconfiguration functionalities: a) Virtual
shared environment
representation
and of the shopfloor,
process perception. combiningdigital
The suggested multiple sensor
world data infrastructure
model and CAD models. The digital
involves shopfloor
three main is rendered a)in Virtual
functionalities: the 3D
environment
representation
Under exploiting
the concept ofthe"Industry
of the shopfloor, capabilities
combining provided
4.0", by sensor
multiple
production Robotprocesses
Operating
data and CAD System
will (ROS)
models.
be pushed framework,
The digital
to b) Semantic
shopfloor
be increasingly representation
is rendered in the 3D
interconnected, of
the world through
environment
information basedtheonimplementation
exploiting athereal
capabilities ofprovided
time basis a unified data
by model
Robot
and, necessarily, for representing
Operating
much System
more the geometrical
(ROS)
efficient. framework,as well ascapacity
the workload
b) Semantic
In this context, state, and
representation
optimization of
c) Dynamic
the
goes world
beyond update
through theofimplementation
the digital
the traditional aim oftwinof based
a unified
capacity on data
real model
time sensor
maximization, for and resource
representing
contributing data comingasfrom
the geometrical
also well the
for organization’s as actual
the shopfloor.
workload
profitability andstate, The
and
value.
communication
c) Dynamic and integration
update of the layer
digital among
twin based theonphysical
real andsensor
time the virtual
and agents is data
resource realized on top
coming of the
from the ROS framework.
actual shopfloor. The
Indeed, lean management and continuous improvement approaches suggest capacity optimization instead of
developed infrastructure
communication is applied
and integration layertoamong
an industrial case and
the physical study
thefrom theagents
virtual automotive industry
is realized on topfocusing on aframework.
of the ROS car’s front axle
The
maximization. The study of capacity optimization and costing models is an important research topic that deserves
assembly. Nevertheless,
developed infrastructureitisis applied
designedtotoan be industrial
generic enough to be able
case study fromtothe
address multipleindustry
automotive cases. focusing on a car’s front axle
contributions from both the practical and theoretical perspectives. This paper presents and discusses a mathematical
assembly. Nevertheless, it is designed to be generic enough to be able to address multiple cases.
model
© 2019 The Authors. management
for capacity based on
Published by Elsevier B.V.different costing models (ABC and TDABC). A generic model has been
© 2019
developed
© 2019
This The
is an Authors.
and
open
The it was Published
used
accessPublished
Authors. to
article underby Elsevier
analyze idle B.V.
capacity
the CC BY-NC-ND
by Elsevier B.V. and to design
license strategies towards the maximization of organization’s
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
This is The
value. an open access article
trade-off capacity under the CC BY-NC-ND
maximization vs license (https://creativecommons.org/licenses/by-nc-nd/4.0/)
operational efficiency is highlighted and it isAgile,
shown that capacity
Peer-review under responsibility of the scientific committee of the International Conference on Changeable,
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Reconfigurable
Peer-review under responsibility of the scientific committee of the International Conference on Changeable, Agile, Reconfigurable
optimization
and Virtual
Peer-review might
Production.
under
and Virtual Production. hide operational
responsibility of the inefficiency.
scientific committee of the International Conference on Changeable, Agile, Reconfigurable
©and2017 TheProduction.
Virtual Authors. Published by Elsevier B.V.
Peer-review under responsibility
Keywords: assembly; of the scientific
digital twin; reconfiguration; committee
mobile of therobot
robot; human Manufacturing
collaboration;Engineering Society International Conference
2017.
Keywords: assembly; digital twin; reconfiguration; mobile robot; human robot collaboration;

* Corresponding
Keywords: author.ABC;
Cost Models; Tel.:TDABC;
+30-2610-910160; fax: +30-2610-997314.
Capacity Management; Idle Capacity; Operational Efficiency
E-mail address: makris@lms.mech.upatras.gr
* Corresponding author. Tel.: +30-2610-910160; fax: +30-2610-997314.
E-mail address: makris@lms.mech.upatras.gr
1. Introduction

The cost of idle capacity is a fundamental information for companies and their management of extreme importance
in modern©production
2351-9789 2019 The Authors. Published
systems. by Elsevier
In general, it isB.V.
defined as unused capacity or production potential and can be measured
This is an open
2351-9789 access
© 2019 Thearticle under
Authors. the CC BY-NC-ND
Published license (https://creativecommons.org/licenses/by-nc-nd/4.0/)
by Elsevier B.V.
in several ways: tons of production, available hours of manufacturing, etc. The management of the idle capacity
Peer-review under responsibility of the scientific committee of the International Conference on Changeable, Agile, Reconfigurable and Virtual
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)
* Paulo Afonso. Tel.: +351 253 510 761; fax: +351 253 604 741
Production.
Peer-review under responsibility of the scientific committee of the International Conference on Changeable, Agile, Reconfigurable and Virtual
E-mail address: psafonso@dps.uminho.pt
Production.

2351-9789
2351-9789©©2017
2019The
TheAuthors.
Authors. Published by Elsevier
Published B.V. B.V.
by Elsevier
Peer-review underaccess
This is an open responsibility
article of the scientific
under committee oflicense
the CC BY-NC-ND the Manufacturing Engineering Society International Conference 2017.
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the International Conference on Changeable, Agile, Reconfigurable
and Virtual Production.
10.1016/j.promfg.2018.12.020
122 Niki Kousi et al. / Procedia Manufacturing 28 (2019) 121–126
2 Kousi et al./ Procedia Manufacturing 00 (2019) 000–000

1. Introduction

The latest trend in today’s globalized market expresses an evident desire to a greater level of product
personalization. However, in current production systems the transition to mass customization generates complexity
and high costs in the design and production [1] that manufacturers need to address in order to maintain their
competiveness and sustainability. The capability of offering more variants per model, and introducing new models
faster, is constrained by the current technologies and the equipment of mass production operations [2].
Achieving flexibility and adaptability that can be defined as the production system's sensitivity to internal and
external changes is regarded as one of the most promising solutions over the last years [3]. However, in current
robot-based production systems, flexibility and reconfiguration are still constrained [4] due to [5]: a) the rigidity of
the used stationary robots, b) the use of fixed and product model dedicated equipment, c) the use of fixed robot
control logic. Mobility both in resources and product level can play a vital role to the realization of such production
concepts as discussed in [6]. In the same work a hybrid and dynamically reconfigurable shopfloor is suggested
employing mobile dual arm workers, namely Mobile Robot Platforms – MRP, and human operators as well as
Mobile Product Platforms (MPP).
The last decades, extensive research has been performed in the field of mobile robotic systems and manipulators
or simple mobile platforms [7]. However, existing applications have limited perception capabilities not allowing real
time adaptation of the system and robot behavior to dynamic environments. Most of the manipulators are restricted
to performing off-line programmed tasks only when they are in fixed positions, thus not fully exploiting their
mobility [6]. On the other hand, digital representation and simulation of the production environment and process
have emerged as a mean of partially handling the optimization of the production system performance [8]. In this era
of digitalization in manufacturing, the Digital Twin concept has gained a lot of attention given the advantages that it
may offer in terms of providing perception and cognition abilities towards more autonomous and intelligent robotic
systems [9] ,[10]. Existing applications of dynamic robot control based on digital modelling and sensor data for
ensuring collision free paths are based on the functionalities provided by Robot Operating System (ROS) framework
[11]. The latter provides a rich content of data types and formats for virtual representing various hardware devices
and multi- sensor data as well as a network of services and topics for broadcasting the captured knowledge.
However, existing infrastructures are not mature enough to support the integration of the discussed hybrid
production paradigm. The involvement of various automated devices such as multiple mobile dual arm workers and
products as well as human operators significantly increases the complexity. To overcome the existing limitations,
this paper introduces a Digital Twin infrastructure able to support the registration of multiple resources as well as
the shopfloor scene real time reconstruction based on multiple sensor data and CAD models.

2. Approach

The abovementioned hybrid production paradigm foresees the use of mobile dual arm workers that can
autonomously navigate inside the shopfloor undertaking multiple assembly operations, such as handling, screwing
etc. while acting as assistants to human operators. To support this assembly paradigm and the autonomy required for
enabling such flexible behavior it is vital to efficiently organize all production entities and reason over the perceived
environment using real time data coming from the shopfloor. To this end, this paper suggests a Digital Twin model
able to virtually represent in real time the shopfloor status providing the following functionalities (Fig. 1.):
• Simplified control integration and sensor data sharing through services. This results to an easier distribution of
acquired data and tasks to all relevant resources such as mobile robots, humans via HMIs etc. (Execution
Coordinator). The integration is realized on top of ROS setting up wrappers for all the involved components.
• Virtual representation of the shopfloor using resource related information (Resource Manager), multiple sensor
data combination (Sensor Manager) and CAD models (Layout Manager). The information is continuously
updated through a network of services by all resources and sensors creating a synthesis of all perception data.
• A unified semantic data model is implemented in order to semantically represent the geometrical as well as the
workload state. This data model should be generic enough to be able to address multiple cases as well as to be
consumed by multiple components inside execution system.
• Real time robot behavior adaption through integration of standard robotic manipulator motion planners and
 Niki Kousi et al. / Procedia Manufacturing 28 (2019) 121–126 123
Kousi et al./ Procedia Manufacturing 00 (2019) 000–000 3

Fig. 1. Digital Twin for real time robot behaviour adaptation.

mobile platform navigations planners to the Digital Twin for reasoning over the shopfloor environment and
generating safe and collision free paths during the execution of the different assembly tasks.

3. Digital Twin implementation

As visualized in Fig. 1. four subcomponents are deployed under the suggested Digital Twin infrastructure: a)
Resource Manager, b) Sensor Manager, c) Layout Manager and d) 3D environment constructor. These software
components were developed based on ROS principles, enabling the communication among network nodes and quick
integration with existing robotic applications. In particular, the described Digital Twin infrastructure has been
deployed and tested on a PC running on Ubuntu 14.04 with ROS indigo version. Each of the included components
are implemented as C++ nodes providing ROS interfaces (topics, services, actions) for exchanging the appropriate
information. As listed in Table 1, the configuration files that are handled and generated from Digital Twin are
following the standard convention of ROS community.
Resource Manager is responsible for registering in the Digital Twin any new resource introduced in the shop
floor. Based on a unified resource data model, the Resource Manager stores all the attributes which compose the
resource model such as maximum payload, minimum velocity, location, transform configuration (.urdf), path
(.yaml) and motion (.srdf) configuration etc. Two sub-components, namely the Resource location monitoring and
Resource status monitoring are responsible for real-time monitoring the status and location of each mobile resource
and update online the actual values in Digital Twin. As visualized in Fig. 2a specific ROS topics and services are
initiated for each subscribed resource broadcasting their real time related data.
The Sensor Manager is responsible for interfacing with the existing sensors’ ROS drivers and registering their
configuration data in Digital Twin repository using the unified data model format. All sensorial data are made
available to robots’ planners, using a publish/subscribe pattern as a communication mechanism as shown in Fig.2b.
Sensors and are uniquely identified by their IDs maintaining a list of respective topics. To facilitate the use of
standard motion and path planning algorithms (e.g. gmapping, amcl, ompl) the 2D-3D sensor data combination
module allows to easily and dynamically (merge multiple, sensor data into one topic.

Table 1. List of file formats used by the Digital Twin’s components.

File format Description
.yaml YAML syntax used for query and set parameters from ROS Parameter Server
.launch An XML file used for easily launching multiple ROS nodes.
.urdf A standard format for descripting robot models, sensors and scenes.
.sdf File format for describing objects and environments for robot simulators like Gazebo.
.srdf Semantic Robot Description Format is the file format used for MoveIt! motion planner configuration
124 Niki Kousi et al. / Procedia Manufacturing 28 (2019) 121–126
4 Kousi et al./ Procedia Manufacturing 00 (2019) 000–000

Fig. 2. a) Resource Manager, b) Sensor Manager, c) 3D Environment constructor & TF tree construction.

To efficiently represent the entire shopfloor, the static layout needs to be represented inside the Digital Twin. The
Layout Manager is responsible for the control and storage of all CADs files related to static fixtures, parts and
products in .sdf format defining also the collisions, the inertia and the mass parameters. This component allows the
user to upload the CAD file and configure various parameters concerning: a) the parts involved in the process, b) the
stationary fixtures included in the shopfloor.
The final component in the process chain is the 3D environment constructor (Fig. 2c). This component retrieves
the locations of all parts, fixtures, sensors and resources to construct an environment with a global world frame
(using ROS Tf - library). Apart from the static parts which their position is defined at the configuration phase there
are also moving objects and obstacles that their position is not well fixed and need to be identified during the
execution. Thus, 3D environment constructor provides an interface with ROS Services and Topic to track and
update the position of all parts inside shop floor based on the real time updates coming from shopfloor sensors.
Fig. 3. represents the unified data model implemented under the deployed infrastructure. The used color coding
maps the consumed data structures by the developed sub-components under the Digital Twin as presented in Fig. 1.

Fig. 3. UML representation of Digital Twin Data Model.

 Niki Kousi et al. / Procedia Manufacturing 28 (2019) 121–126 125
Kousi et al./ Procedia Manufacturing 00 (2019) 000–000 5

Fig. 4. Generation of a) path plan and b) motion plan based on Digital Twin provided data, c) MRP H/W configuration for the automot ive case.

A lot of research has been done related to the avoidance of collisions among resources and unmapped obstacles
inside the shopfloor environment. Exploiting existing algorithms, the Digital twin provides interfaces with robot’s
path and trajectory planners, to achieve online re-planning based on fused real-time information from shopfloor.
The path planning component interface that was implemented is based on ROS navigation stack for mobile
robots. In particular the Digital Twin provides: a) The transformation for every coordinate frame of a robot which is
described in a .urdf file, b) distance sensors and odometry information, that is required for mapping and localization
of the resource as well as for avoiding obstacles, c) the global map of the shopfloor – 3D map created in occupancy
grid using OctoMap library and the costmap with obstacles information d) the configuration of the planner.
For the motion planning of robotic arms, the Digital Twin model facilitates an interface with the MoveIt!
framework. Three kinds of information are needed for the setup the motion planning: a) robot’s urdf file, b) robot’s
.srdf file and c) MoveIt! configuration files for joint limits, kinematics, motion planning, perception input and other
information. For ensuring a collision free trajectory of the robot, the Digital Twin provides global planning scene
with all the objects and resources. This planning scene is published as an occupancy map constructed by the 3D
environment constructor. This map is then online updated using the available multi - sensor data.

4. Case study

The proposed system has been implemented and tested in a case study from the automotive sector for the
assembly of a vehicle’s front axle. The hybrid production scenario considered was based on the analysis performed
by Kousi et al. [6] involving one MRP, one MPP, one compression machine and one human operator for the
compression and assembly of vehicle’s right damper on the axle’s disk. For the efficient execution of the scenario,
the MRP needs to perceive the: a) real time damper and moving fixture location to compensate localization errors
that cannot foreseen offline, using rc_visard stereo cameras data, b) static obstacles and moving humans / obstacles
for ensuring collision free navigation, using 2D laser scanner data. Direct interaction with the human when the latter
needs to online “instruct” the robot is also available through the Kinect provided skeleton recognition library.

Fig. 5. Automotive case study Digital Twin visualization.

126 Niki Kousi et al. / Procedia Manufacturing 28 (2019) 121–126
6 Kousi et al./ Procedia Manufacturing 00 (2019) 000–000

The Digital Twin infrastructure was deployed towards integrating and real time updating the virtual world with
the information coming from the damper, obstacles and human detection modules broadcasting them to the robot
motion and path planners. To test the functionality of the developed system, the scenario has been set up in
GAZEBO physics simulation engine (Fig. 5) using: a) CAD files of LMS Machine Shop for the layout
representation, b) ROS virtual controllers of Universal Robots (UR) arms and mobile platform, c) 2D laser scanner
data for the 2D map creation d) simulated stereo camera data for virtual object detection and e) real Kinect data
mapping a physical Kinect to the physical one for testing the Human Robot Interaction through gestures recognition.

5. Conclusions & Future Work

The latest trends in EU manufacturing foster the deployment of hybrid production systems where humans can co-
exist and cooperate with mobile multi-arm robots. Such flexible robot workers should be able to perceive their
environment in terms of process requirements and human activity. Driven by this need, the Digital Twin system
presented in this work, provides the infrastructure for integrating all the hardware components involved in the
assembly and synthesizing all the data coming from the shopfloor under a unified common environment. Upon this
model, each resource applies cognition techniques to transform the sensor data into usable information and
eventually to knowledge of the shop floor status. The developed system was deployed in a case study from the
automotive sector. The unified data model however is generic enough so as to support multiple cases including
mobile and/or stationary robots, single and / or multi arm manipulators, multiple sensor types (2D, 3D, F/T etc.) and
human operators. The deployment of the Digital Twin allows the production system to compensate the real-world
uncertainty of the environment as well as of human behavior unpredictability. Thus, the need for offline programing
and validation of robot motions when process changes are required is not needed, minimizing the needed time and
effort. However, it such a system exhibits higher complexity due to the increased amount of data captured and
processed. Future work will involve integration of the Digital Twin with: a) the physical robotic set up to validate its
performance and b) high-level decision-making mechanisms allowing the reconfiguration of the system in shopfloor
level through task re-allocation based on the real time production needs (new product variants etc.).

Acknowledgements

This work has been partially funded by the EC research project “THOMAS – Mobile dual arm robotic workers
with embedded cognition for hybrid and dynamically reconfigurable manufacturing systems” (Grant Agreement:
723616) (www.thomas-project.eu/).

References

[1] G. Chryssolouris, Manufacturing Systems: Theory and Practice. second ed. Springer-Verlag, New York (2006).
[2] S. Giordani, M. Lujak, F. Martinelli, A distributed multi-agent production planning and scheduling framework for mobile robots,
Computers and Industrial Engineering, 64 (2013) 19–30.
[3] J. Krüger, L. Wang, A. Verl, T. Bauernhansl, E. Carpanzano, S. Makris, J. Fleischer, G. Reinhart, J. Franke, S. Pellegrinelli, "Innovative
Control of Assembly Systems and Lines", CIRP Annals of Manufacturing Technology, 66 (2018) 707-730
[4] G. Michalos, N. Kousi, S. Makris, G. Chryssolouris, Performance Assessment of Production Systems with Mobile Robots. Procedia CIRP,
41 (2016) 195–200.
[5] G. Michalos, S. Makris, N. Papakostas, D. Mourtzis, G. Chryssolouris, Automotive assembly technologies review: challenges and outlook
for a flexible and adaptive approach. CIRP Journal of Manufacturing Science and Technology, 2 (2010) 81-91.
[6] N. Kousi, G. Michalos, S. Aivaliots, S. Makris, An outlook on future assembly systems introducing robotic mobile dual arm workers, 51st
CIRP Conference on Manufacturing Systems, (CMS 2018), 16-18 May, Stockholm, Sweden, (2018), to be published in Procedia
[7] N. Kousi, S. Koukas, G. Michalos, S. Makris, “Scheduling of smart intra – factory material supply operations using mobile robots”,
Accepted for publication on International Journal of Production Research (2018).
[8] J. Váncza, László Monostori, Cyber-physical Manufacturing in the Light of Professor Kanji Ueda's Legacy, 50 th CIRP Conference on
Manufacturing systems Procedia CIRP, 63 (2017) 631-638
[9] G. Chryssolouris, D. Mavrikios, N. Papakostas, D. Mourtzis, G. Michalos, K. Georgoulias, Digital manufacturing: History, perspectives,
and outlook, Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 223 (2009) 451 – 462.
[10] S. Makris, G. Michalos, G. Chryssolouris, Virtual Commissioning of an Assembly Cell with Cooperating Robots, Advances in Deci sion
Sciences 2012, Article ID 428060, (2012).
[11] URL Robot Operating System http://www.ros.org/, last accessed 02/06/2018