IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 12, NO.

2, APRIL 2015

769

Toward Welding Robot With Human Knowledge:

A Remotely-Controlled Approach
YuKang Liu and YuMing Zhang

AbstractThis paper presents a remotely controlled welding

scheme that enables transformation of human welder knowledge
into a welding robot. In particular, a 6-DOF UR-5 industrial robot
arm is equipped with sensors to observe the welding process,
including a compact 3D weld pool surface sensing system and
an additional camera to provide direct view of the work-piece.
Human welder operates a virtual welding torch, whose motion is
tracked by a Leap sensor. To remotely operate the robot based
on the motion information from the Leap sensor, a predictive
control approach is proposed to accurately track the human
motion by controlling the speed of the robot arm movement.
Tracking experiments are conducted to track both simulated
movement with varying speed and actual human hand movement.
It is found that the proposed predictive controller is able to track
human hand movement with satisfactory accuracy. A welding
experiment has also been conducted to verify the effectiveness of
the proposed remotely-controlled welding system. A foundation is
thus established to realize teleoperation and help transfer human
knowledge to the welding robot.
Note to Practitioners Welding is the nal stage for manufacturing of many high value added products. While industrial
welding robots have been in use for several decades, they are
preprogrammed actuators with limited, if any, intelligence. Given
that manufacturing is moving towards more customized production, next-generation welding robots that can intelligently adjust
to various welding tasks are urgently needed. Unfortunately,
equipping robots with human knowledge is challenging. This
paper establishes a teleoperation system that can rapidly transform human welder knowledge into welding robots. A predictive
control approach is developed to accurately track the human hand
movement by controlling the speed of the robot arm movement. A
foundation is thus established to mechanize the human welder's
knowledge and develop next-generation welding robots.
Index TermsMotion tracking, predictive control, robot arm,
teleoperation, welding.

I. INTRODUCTION
Current industrial welding robots are articulated arms with a preprogrammed set of movement, which lack the skills human welders
possess. Their applications are thus mostly limited to assembly lines
for mass-produced products, such as automobiles. However, the trend
of manufacturing is to produce customized products in small batches,
Manuscript received March 05, 2014; revised July 29, 2014; accepted
September 08, 2014. Date of publication October 08, 2014; date of current version April 03, 2015. This paper was recommended for publication by Associate
Editor D. Song and Editor J. Wen upon evaluation of the reviewers' comments.
This work was supported in part by the National Science Foundation under
Grant CMMI-0927707 and Grant IIS-1208420.
The authors are with the Institute for Sustainable Manufacturing and Department of Electrical Engineering, University of Kentucky, Lexington, KY 40506
USA (e-mail: yuming.zhang@uky.edu).
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TASE.2014.2359006

where ideal automated production lines are not cost effective. Intelligent welding robots that possess knowledge comparable to skilled
welders but with less physical restrictions and can work in harsh environments will thus be one of the keys to maintaining a competitive
manufacturing industry despite relatively high labor costs/wages.
Although welding process sensing and control have been widely
studied for decades [1][4], such control methods typically involve two
steps: modeling of the welding process and design of the control algorithm. An alternative which has been proved to be more direct and
effective in many other applications is to learn from human operators
such that the design of the control algorithm becomes a one step process
[5]. Modeling human welder response and transferring the resultant intelligent model to the welding robot thus would provide a convenient
method to take advantage of valuable human welder experience. It also
paves the foundation to improve and exceeding human physical limitations. Hence, in this paper, the authors introduce a new human machine
welding paradigm, virtual welding, that can transfer human knowledge to welding robots which may work in harsh environments.
In the proposed framework, a welding robot working in the welding
environment is augmented with sensors to observe the work-piece and
reconstruct the 3D weld pool. The obtained data from the sensors as
feedback from actual welding process is viewed by a skilled welder
in a virtual welding environment. The welder then adjusts the hand
movement accordingly. The data and adjustments will also be recorded
and analyzed to model how the welder responds to the 3D weld pool
surface.
As such, virtual welding can be operated at three different levels: (a)
remotely controlled welding (i.e., teleoperation, where welder's motion and welding process information will be recorded and utilized to
build a welder response model for autonomous welding); (b) cooperative welding (human welder mostly monitors the process carried
out by the robot and occasionally make adjustment if he/she needs
to); and (c) autonomous welding (robot runs in a fully automatic way
under closed-loop control with knowledge distilled from skilled human
welder). Among all three levels, teleoperation serves as the foundation
and is discussed in this paper.
Through remotely controlled welding (or teleoperation) [6], [7], a
welder can remotely operate the welding robot, which would enable
welding in hazardous and less structured environments. However, current remote welding teleoperations are mostly for seam tracking applications. Up to now there are few literatures that are considered as
a systematic way of transferring human knowledge into the welding
robot. Among various devices to control robots [8][11] (e.g., joystick,
mouse, keyboard, haptic devices, touch-screen). The Leap sensor [11]
can sense motion with an accuracy of hundredth of a millimeter and has
found its applications in various elds, including gaming industries,
surgery, architecture, and motion tracking. Compared to other devices,
using a virtual welding torch (similar in size and weight of an actual
welding torch) and leap sensor to track its position appears to be a more
natural way for human welders to perform teleoperation. In this study,
the leap motion sensor will be adopted to sense the tool position of the
virtual welding torch and control the robot based on the sensed coordinates. Details of the Leap sensor will be provided in Section II.
In this paper, a remotely controlled robot welding scheme is proposed to enable the transformation of the human welder knowledge

1545-5955 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

770

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 12, NO. 2, APRIL 2015

Fig. 1. Teleoperation system schematic and information ow.

into the welding robot. The welding robot is equipped with sensors to
monitor the welding process. The position of the robot tool movement
is closed-loop controlled by minimizing the cost function consisting of
the position errors and amplitude of the speed. Tracking experiments
are conducted to test the proposed predictive control algorithm. The
welding experiment is further carried out to verify the effectiveness of
the proposed teleoperation system. The remainder of this paper is organized as follows. In Section II, an overview of the teleoperation system
is provided. In Section III, the predictive control algorithm is derived.
Tracking experiments are conducted to track simulated signals with
varying speeds in Section IV and the results obtained by the proposed
scheme are compared to open-loop control. In Section V, human hand
movement is tracked and the tracking experiment results are presented
and analyzed. A welding experiment is conducted and results are presented in Section VI. Conclusions are nally drawn in Section VII.
II. TELEOPERATION SYSTEM OVERVIEW
The proposed teleoperation system schematic is illustrated in Fig. 1.
In the virtual station, a human welder can view the monitor where the
weld pool images are displayed, and moves the virtual welding torch
[Fig. 2(c)] accordingly as if he/she is right in front of the work-piece.
The human welder movement is accurately captured by a Leap sensor,
and the obtained 3D coordinates will be sent to the PC. The robot arm
is equipped with the welding torch and compact weld pool sensing
system. It receives commands via Ethernet from the PC, including the
next robot tool pose (i.e., 3D positions and orientations), and robot tool
movement speed. It then executes the command and sends the current
robot tool pose back to the PC. It is noticed that the system telecommunication delay caused by sending and receiving packages between the
robot and PC is smaller than 20 ms most of the time. Thus, the system
telecommunication delay can be safely neglected in our system.
The robot utilized in this study is Universal Robot UR-5 with 6 Degree of Freedom [Fig. 2(a)]. The accuracy of the robot repeatability
is 0.1 mm. Fig. 2(b) shows the developed compact 3D weld pool surface sensing system for our robotic welding system. A low-power laser
(19 by 19 structure light pattern) is projected to the weld pool surface
and its reection from the specular weld pool surface is intercepted by
an imaging plane and imaged by a CCD camera [camera 1 in Fig. 2(b)].
It is known that arc light is an omnidirectional light source. Its intensity
decreases quadratically with the distance traveled, but the laser, due to
its coherent nature, does not signicantly lose its intensity. Hence, it
is possible to intercept the reection of the illumination laser from the
weld pool surface with an imaging plane placed at an appropriate distance from the arc. From the distorted reection pattern on the imaging
plane and the assumption of a smooth weld pool surface, the 3D shape
of the weld pool surface can be obtained. By using specic image processing and reconstruction algorithm [12], 3D specular weld pool can
be reconstructed in real-time. The proposed teleoperation system operates at a frame rate of 10 Hz, which is considered real-time in our ap-

Fig. 2. UR-5 robot arm and sensing system. (a) UR-5 robot arm with compact
3D weld pool surface sensing system attached to the robot tool. (b) Detailed
view of the compact sensing system. (c) Virtual welding torch.

plication. In addition, camera 2 [in Fig. 2(b)] can provide an eye view
similar to what welders can see during the welding process.
The Leap sensor is an advanced motion sensor which is utilized in
this study to accurately capture the human welder movement. Fig. 3 depicts the Leap Motion Sensor. Using two cameras and three infrared
LEDs, the sensor observes a roughly hemispherical area to a distance
of about 1 m. It can track ngers or similar items to a spatial precision of
0.01 mm [11]. Fig. 4 plots a sample human welder movement along the
axis. It is observed that the ne human welder's hand movement can
be tracked by the sensor. Although it is intended to move only in one
direction, on the other two directions slight movement is also observed.
In our application which is at plate welding, only the movement along
the welding direction (or axis in this study) is concerned. Movement
along the other two axes will be thus considered as noises and will be
ignored as follows in the paper. It is noticed the proposed teleoperation
system can be generalized to learn human motion for other applications
(e.g., grasping and assembling), as long as a process feedback can be
obtained.
III. PREDICTIVE CONTROL OF ROBOT MOVEMENT
For welding applications, it is preferred that the welding robot maintains a smooth and accurate tracking of the welder's hand movement.
Model Predictive Control (MPC) has received considerable attention
in past decades due to its robustness with respect to model uncertainty
[13][16]. In this section, the predictive control scheme for accurately
tracking the human hand movement is detailed. The output of the control system is the robot position, and the control input is the robot
speed.

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 12, NO. 2, APRIL 2015

771

Fig. 3. Leap motion sensor.

Fig. 6. Filtered hand movement speed using the optimal smoothing coefcient
obtained in Fig. 5.

Fig. 4. A sample human welder movement along the axis tracked by Leap
motion sensor. Two straight horizontal solid lines indicate desired and axis
movement, respectively.

Fig. 7. Five-step-ahead prediction error for hand movement in Fig. 4.

Fig. 5. Average direction prediction error with respect to smoothing coefcient for human movement in Fig. 4.

A. Prediction of the Human Hand Movement

In predictive control [13], a reference signal is needed to compute the
control actions. In our study, the reference signal is the human hand
movement. At instant , the controller needs to determine the speed
based on the feedback
to drive the robot to track human hand
movement .
should be large enough to achieve a roThe prediction horizon
bust control. However, the settling time increases as increases. It is
can achieve relafound from tracking experiment trials that
tively short settling time and good robustness. In our application, the
is dened as
desired trajectory

and the ve-step-ahead prediction error reaches the minimum when

. Fig. 6 plots the ltered speed using the optimal smoothing
coefcient. In this particular movement, the hand movement generally
becomes slower, as can be reected from the ltered speed changing
from 28 mm/s to about 15 mm/s.
Because the smoothness of the human hand movement varies from
person to person, it is evident that different operators should have different smoothing coefcients. To obtain the smoothing coefcient for
a specic operator, a training period can be conducted and process described in this section can be applied accordingly.
Fig. 7 depicts the ve-step-ahead prediction performance using
equation (4). It is observed that even for relatively large speed variations in Fig. 4, the prediction errors remain less than 2 mm for most
of the time. In the following section, this prediction will be utilized in
the predictive control algorithm.
B. Predictive Control Algorithm
In this section, predictive control of robot motion for teleoperation
is derived.
based
At instant , the controller needs to determine the speed
to drive the robot to track human hand movement
on the feedback
.
The following one-step-ahead prediction equation can be obtained:
(2)

(1)
where
is the sampling time,
is the ltered speed at instant
, and
is the smoothing coefcient. As becomes larger,
the system will track the set point with slower speed but better robustness and smoothness. Fig. 5 illustrates ve-step-ahead prediction
error for human hand movement specied in Fig. 4, with respect to
smoothing coefcient from 0 to 1. It is observed that
can achieve a good tradeoff between response speed and robustness,

Suppose the future control action is constant, i.e.

,
, where
is the prediction horizon, -step-ahead prediction
yields
(3)
The prediction equation can be further expressed in matrix form
(4)

772

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 12, NO. 2, APRIL 2015

where
..
.

..
.

..
.

Large robot movement speed may generate nonsmooth robot movement and shaking, which is not preferable in our application. Thus, the
control objective should minimize the tracking error as well as the robot
speed. The following cost function is used:
Fig. 8. Tracking simulated varying speed signal using open-loop and closedloop control.

(5)
where
.
penalizes the magnitude of the control input
The coefcient
(i.e., robot movement speed). If is relatively larger, smoother robot
movement is obtained, but the tracking accuracy is decreased. If is
smaller, tracking performance is better, but it may cause un-smoothness of robot movement due to large movement speed. In this study,
is chosen, which implies that an error of 1
mm in the position has the same contribution to the cost function as
robot speed change of 1 mm/s. Tracking experiments described in
Sections IV and V conrm that the selected is able to generate satisfactory tracking performance and maintain a smooth robot movement.
The control law is calculated such that
(6)
Equation (6) can be further expressed as (7) shown at bottom of page.
The predictive control law is nally expressed as

(8)
Equation (8) is an analytical solution to the optimization of the cost
function specied in (5), and can be implemented in real-time.
IV. TRACKING SIMULATED SIGNAL
To validate the derived predictive control law and its advantages over
open-loop robot teleoperation, tracking experiments are conducted to
track signals generated by the computer program. With x position from
412 mm to 432 mm, the command speed is 0.001 m/s, from 432 mm to
452 mm the command speed is 0.005 m/s. From 452 mm to 472 mm,
472 mm to 492 mm, and 492 mm to 512 mm, the command speed is
set at 0.003, 0.007, and 0.006 m/s, respectively. Equation (8) is utilized to calculate the robot speed signal, which will be inputted into

Fig. 9. Control input calculated by the proposed predictive controller.

the teleoperation system. Fig. 8 illustrates the tracking performance

for open-loop control with 0.005, 0.015 m/s, and closed-loop control.
It is observed that with open-loop control of 0.005 m/s robot speed,
large tracking errors are accumulated during the robot movement, with
the largest error being 24 mm, which is considered unacceptable in
our application. For open-loop control with 0.015 m/s robot speed,
good tracking performance is achieved. However, because the robot is
moving at a relatively large speed even for 0.001 m/s command speed,
large oscillations are frequently observed. For the closed-loop control,
on the other hand, the performance is comparable to open-loop control
of 0.015 m/s, yet minimum robot speed is applied to minimize the robot
oscillation. Fig. 9 plots the calculated robot speed in this experiment.
It can be seen that for a different command signal, the robot speed is
different. This makes sense because for slow human hand movement
(for example, from 0 to 18 s), relatively small speed can be applied to
the system to obtain acceptable tracking performance. For quick human
hand movement, however, large speed should be applied to the robot
in order to obtain acceptable performance.
V. TRACKING HUMAN HAND MOVEMENT
In this section, tracking performance is tested on actual human hand
movement. Human hand movement captured by the Leap sensor described in Section II is tracked by the robot and the performance is
analyzed. As can be seen from Fig. 10, the human hand movement is

(7)

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 12, NO. 2, APRIL 2015

773

Fig. 10. Tracking human hand movement using proposed closed-loop predictive control.

Fig. 12. Welding experiment inputs and weld pool parameters. (a) Welding
current and voltage. (b) Measured weld pool characteristic parameters.

Fig. 11. Control input calculated by the proposed predictive controller.

TABLE I
WELDING PARAMETERS

constantly varying. The trend of this particular movement is that the

movement is relatively fast from 0 to 9 s, slows down from 9 to 15 s,
and speeds up again from 15 to 23 s. Fig. 10 illustrates the tracking
performance using the proposed closed-loop predictive controller. It is
observed that the tracking errors are kept minimal and is acceptable in
our application. Fig. 11 plots the control signal. Although the calculated control signal is varying, it generally coincides with the trend of
the hand movement captured by the sensor.
VI. WELDING EXPERIMENT
The teleoperation experiment of the welding robot is conducted in
this section and the experimental results are presented/analyzed.
In the experimental system, a stainless steel plate is welded using
the direct current electrode negative gas tungsten arc welding (GTAW)
process. The thickness of the plate is 2 mm. The welding parameters are
illustrated in Table I. The welding current is randomly changed from 25
to 40 A to generate a uctuating weld pool surface, and the arc length is
maintained at 4 mm. The human welder observes the weld pool based
on camera 2 feedback and control of the virtual welding torch (welding
speed is thus controlled by the human welder). Camera 1 is utilized to
reconstruct the 3D weld pool surface and display it in the monitor for
a human to view as well.
Fig. 12(a) plots the welding current and voltage in this experiment.
Fig. 12(b) depicts the measured weld pool surface characteristic parameters (detailed denition can be seen in [16]), that is the weld pool
width, length, and convexity (average height of the weld pool surface).
Fig. 13 shows three sample images captured from camera 2 (at instant
,
, and
, respectively), corresponding images captured from camera 1, and reconstructed 3D weld

pool surface. Human welder observes images from camera 2 on the

monitor, and adjusts his/her hand movement accordingly. The reconstructed weld pool surface (characterized by its width, length, and convexity as the model inputs), and the welding movement speed (as the
model output) can then form a welder response model, which can be
transferred to the welding robot to perform automated welding.
At instant t1, the current (about 32 A) is relatively small. As can be
observed in Fig. 12(b), at this instant the measured weld pool length,
width, and convexity are 5.8, 4.1, and 0.12 mm, respectively. From
Fig. 13(a), the width and length can be inferred, yet the convexity
cannot be obtained from this eye view. In Fig. 13(b), the reected laser
pattern is shown and Fig. 13(c) the reconstructed weld pool is rendered.
From the reconstructed weld pool surface, the welder can get a detailed
view about the pool shape, and the geometrical parameters including
the width, length, and convexity can also be conveniently obtained.
For the other two instants different weld pool shapes are observed, corresponding to varying weld pool characteristic parameters shown in
Fig. 12(b).
VII. CONCLUSION
This paper introduced a new human machine welding paradigm,
virtual welding, which can transfer human knowledge to welding
robots. To facilitate this new paradigm, a predictive robot arm teleoperation scheme is proposed. In this framework, the motion of the
human welder movement is tracked accurately by the Leap sensor,
and a 6 DOF UR-5 industrial robot arm is augmented with sensors
to directly observe the work-piece (eyeview) and reconstruct the 3D
weld pool surface. The human hand movement is then closed-loop
tracked by minimizing the cost function consisting of the position errors and amplitude of the speed. The derived predictive control algorithm is in closed-form to enable a real-time implementation. Tracking
experiments are conducted to track simulated movement with varying
speed and human hand movement. The welding experiment is further conducted to verify the effectiveness of the proposed teleoperation system. It is found that the proposed remotely controlled robotic
welding system can accurately track the human welder movement and
record the movement/3D weld pool surface parameters accordingly. A
foundation is thus established to systematically transfer the human intelligent into the welding robot.
Future work will explore tracking welder's motion for welding applications with complex 3D shape. In addition, teleoperation experiments

774

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 12, NO. 2, APRIL 2015

Fig. 13. Original images and reconstructed weld pool surface at instant t1t3. (a) Image captured by camera 2. (b) Image captured by camera 1. (c) Reconstructed
weld pool surfaces.

will also be conducted by skilled human welders and their responses to

3D weld pool surfaces will be learned and transferred to the welding
robot to perform automated welding.

REFERENCES
[1] A. C. Guu and S. I. Rokhlin, Arc weld process control using radiographic sensing, Mater. Eval., vol. 50, no. 11, p. 1344, 1992.
[2] H. Matsui, T. Chiba, and K. Yamazaki, Detection and amplication
of the molten pool natural oscillation in consumable electrode arc
welding, Welding Int., vol. 28, no. 1, pp. 512, 2014.
[3] P. Ghanty, M. Vasudevan, and D. P. Mukherjee et al., Articial neural
network approach for estimating weld bead width and depth of penetration from infrared thermal image of weld pool, Sci. Technol. Welding
Joining, vol. 13, no. 4, pp. 395401, 2008.
[4] Y. K. Liu and Y. M. Zhang, Model-based predictive control of weld
penetration in gas tungsten arc welding, IEEE Trans. Control Syst.
Technol., vol. 22, no. 3, pp. 955966, May 2014.
[5] Y. K. Liu, W. J. Zhang, and Y. M. Zhang, Dynamic neuro-fuzzy
based human intelligence modeling and control in GTAW, IEEE
Trans. Autom. Sci. Eng., 2013, to be published.
[6] R. Bogue, Robots in the nuclear industry: A review of technologies
and applications, Ind. Robot: An Int. J., vol. 38, no. 2, pp. 113118,
2011.
[7] H. T. Chen, H. C. Li, H. M. Gao, L. Wu, and G. J. Zhang, Optimal
digital ltering for tremor suppression in a master-slave robot remote
welding system, Robotic Welding, Intell. Autom., pp. 8185, 2011.

[8] S. Lee and G. S. Sukhatme et al., Haptic teleoperation of a mobile

robot: A user study, Teleop. Virtual Environ., vol. 14, no. 3, pp.
345265, 2005.
[9] S. Cossell, M. Whitty, and J. Guivant, Streaming Kinect data for robot
teleoperation, in Proc. Australasian Conf. Robot. Autom., Melbourne,
Australia, Dec. 79, 2011.
[10] Y. H. Seo and H. Y. Jung et al., Remote data acquisition and
touch-based control of a mobile robot using a smart phone, Commun.
Comput. Inf. Sci., vol. 265, pp. 219226, 2012.
[11] T. Daniel, Leap motion: 3D hands-free motion control, unbound.
May 20, 2012. [Online]. Available: http://news.cnet.com/830111386_3-57437404-76/leap-motion-3d-hands-free-motion-controlunbound/
[12] W. J. Zhang, Y. K. Liu, X. Wang, and Y. M. Zhang, Characterization
of three-dimensional weld pool surface in GTAW., Welding J., vol.
91, no. 7, pp. 195s203s, 2012.
[13] S. J. Qin and T. A. Badgwell, A survey of industrial model predictive
control technology, Control Eng. Practice, vol. 11, no. 7, pp. 733764,
2003.
[14] S. J. Yoo, Y. H. Choi, and J. B. Park, Generalized predictive control
based on self-recurrent wavelet neural network for stable path tracking
of mobile robots: Adaptive learning rates approach, IEEE Trans. Circuits Syst., vol. 53, no. 6, pp. 13811394, Jun. 2006.
[15] S. S. Ge, Z. Li, and H. Yang, Data driven adaptive predictive control
for holonomic constrained under-actuated biped robots, IEEE Trans.
Control Syst. Technol., vol. 20, no. 3, pp. 787795, May. 2012.
[16] Y. K. Liu and Y. M. Zhang, Control of 3D weld pool surface, Control
Eng. Practice, vol. 21, no. 11, pp. 14691480, 2013.