C.-H. Chen, C. Cheng, D. Page, A. Koschan, and M.

Abidi, "Tracking a Moving Object with Real-time Obstacle Avoidance,"

240 International Journal of Industrial Robot, Special Issue on Robot Control and Programming, Vol. 33, No. 6, pp. 460-468, 2006.

Research article

Tracking a moving object with real-time

obstacle avoidance
Chung-Hao Chen, Chang Cheng, David Page, Andreas Koschan and Mongi Abidi
Imaging, Robotics and Intelligent Systems Lab, Department of Electrical and Computer Engineering, The University of Tennessee,
Knoxville, Tennessee, USA

Abstract
Purpose Aims to develop a robotic platform to autonomously track a moving object
Design/methodology/approach This robotic platform, based on a modular system known as SafeBot, uses two sensors: a visual CCD camera and a
laser-based range sensor. The rigidly mounted camera tracks an object in front of the platform and generates appropriate drive commands to keep the
object in view, even if the object itself moves. The range sensor detects other objects as the platform moves to provide real-time obstacle avoidance
while continuously tracking the original object.
Findings The current approach successfully tracks an object, particularly a human subject, and avoids reasonably sized obstacles, but on-board
processing limitations restrict the speed of the object to approximately 5 km/h.
Originality/value The core technology a moving object tracked by a mobile robot with real-time obstacle avoidance is an integrated system
comprising object tracking on a mobile platform and real-time obstacle avoidance with robotic control. This system is applicable to a variety of
automated applications such as inventory management, industrial palette distribution, and intruder surveillance.

Keywords Computer applications, Video equipment

Paper type Research paper

Introduction unit that is self sufficient and independent in terms of

In todays world, an autonomous robotic platform equipped
with various kinds of sensors is becoming more widely used
for inspecting objects, monitoring areas, and assisting
humans. The imaging, robotics, and intelligent systems
(IRIS) Laboratory at the University of Tennessee (UT) has
developed a robotic platform, known as SafeBot, to augment
the missions of current and future autonomous robotic
platforms (Plate 1). We propose SafeBot as a robotics system
that incorporates four sensors in a modular configuration.
These sensors include a laser range sensor brick, a thermal
sensor brick, a visual sensor brick, and a nuclear sensor brick.
The term brick is used as an analogy to emphasize that the
modularity of the robotics platform where the proposed
paradigm involves stacking appropriate bricks together to
increase functionality or capability. This paradigm is in
contrast to the traditional method where the subsystems,
particularly the sensors, of a robotics platform are highly
interconnected and interwoven, and thus the addition of new
sensors or capabilities severely impacts the design if such
upgrades are even feasible. Each sensor brick is a modular
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0143-991X.htm
processing and communications needs. The bricks come
together and work as a unit via a wireless network. IRIS Lab
research seeks to incorporate autonomous intelligence into
these sensors through the development of fusion, processing,
and control algorithms.
The application that we focus on in this research is a low
profile platform that is able to follow a human target
autonomously. As a result, the thermal sensor brick and nuclear
sensor brick are not used in this application and thus are not
discussed in this paper. Figure 1 shows a potential task where
usually an individual uses a hand truck to move palette
inventory throughout an industrial plant. The SafeBot platform
is currently able to carry a payload of approximately 150 kg, and
the geometrical configuration is about the dimensions of a
standard palette. The goal is for SafeBot to carry a palette and
then follow a human. To achieve this objective, SafeBot uses the
visual brick to track the human and the laser range brick to avoid
obstacles. The human operator essentially serves as the global
path planner a highly complicated skill without concern for
the labor of lifting the palette or for the local path planning to
avoid bumping the palette into obstacles. The SafeBot
autonomously follows the operator and takes care of itself. To
realize this system, the research in this paper develops tracking
and obstacle avoidance technologies within the brick paradigm.

Industrial Robot: An International Journal

33/6 (2006) 460 468 This work is supported by the University Research Program in Robotics
q Emerald Group Publishing Limited [ISSN 0143-991X] under grant DOE-DE-FG52-2004NA25589 and by the DOD/
[DOI 10.1108/01439910610705635] RDECOM/NAC/ARC Program under grant W56HZV-04-2-0001.

460
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Plate 1 Example application for the SafeBot platform
SafeBot overview
The SafeBot modular robotics system, consists of five
component bricks (Figure 1):
1 sensor bricks (different types of sensors such as a vision
sensor, the thermal sensor, the laser range sensor, the
nuclear sensor, etc.);
2 a mobility brick;
3 a processing and intelligence brick;
4 a network routing brick; and
5 a human interface brick.
Each brick component has its own key task and can act as an
independent or cooperating unit. The design of the system
allows upgrading or overhauling the system modularly by
adding or removing bricks as needed. This flexibility means
that failed components are readily replaced by swapping out
appropriate bricks. In addition, the system must be functional
in both generic and specific applications. Figure 1 shows the
modular robotic system architecture.
In terms of a sensor brick component, it does not only
represent a particular sensor, but each sensor brick is a plug-
and-play embedded computer. Its functions include
Figure 1 The block diagram shows the modular robotics system architecture
461
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
controlling the generic algorithms and other software
components contained in the brick such as image
processing, path planning, storage control, object tracking,
and so on. Currently, we propose four sensor bricks. First, the
laser range sensor brick, which is mainly used for autonomous
navigation and 3D scene modeling. Second, the thermal
sensor brick, which captures infrared images. Third, the CCD
visual sensor brick, which acquires visual images. Finally, the
nuclear sensor brick, which is a neutron and gamma ray
detector. Figure 2 shows these sensor bricks and the SafeBot
platform. The human interface is a connection between the
human and the robot that consists of graphical interface
running on a laptop computer. The network routing brick
exchanges data between bricks using a wireless transmission.
The mobility brick, as the name implies, enables the
movement of the robot.
Object tracking with obstacle avoidance
The initial SafeBot concept has been introduced in the
context of automotive inspection (Qian et al., 2005), but in
this paper, we extend the SafeBot and brick architecture to
the industrial application of palette manipulation and to the
surveillance application of following a human suspect. With
this latter application, most surveillance systems require that
the desired object, i.e. a human suspect, stay within the
surveillance range of the system (Kang et al., 2004; Haritaoglu
et al., 2000). If the object goes beyond this range, the system
can no longer track the object. As a result, the surveillance
system lacks flexibility. An alternative solution the one
proposed in this paper is to send a mobile robotics platform
with sensors (Schulz et al., 2001; Philomin et al., 2000; Perez
et al., 2004), such as video cameras to track the moving
object. NASA proposed a similar concept (Graham and
Shillcutt, 2003), but they use it for assisting astronauts to
carry more equipment in space exploration tasks. These
systems are primarily concerned with object tracking, and are
not concerned with the obstacle avoidance problem. On the
other hand, if there is an obstacle sitting in the way of
the mobile robot, it must avoid the obstacle while still tracking
the desired object at the same time.
Therefore, the motivation for this work stems from the need
to track a moving object by a mobile robotics platform while
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Figure 2 Examples of IRIS Lab sensor bricks
simultaneously avoiding obstacles in real-time. A common
obstacle avoidance approach is the potential fields method
(Barraquand and Latombe, 1991; Koren and Borenstein,
1991; Borenstein and Koren, 1989; Latombe, 1991). With
the potential fields method, an artificial field is estimated such
that a goal position generates an attractive force to the robot
and each obstacle generates a repulsive force on the robot.
The resultant force determines the moving direction of the
robot. Unfortunately, this approach is inefficient if the goal
in our case the tracked object is dynamic and continuously
changing its position. Therefore, the contributions of this
paper are not only to construct an object tracking robotic
platform but also to modify the potential fields method to
enable real-time obstacle avoidance.
The rest of this paper is organized as follows. Section 2
discusses the core technology of the object tracking robot,
which includes the object tracking strategy, the obstacle
detection and avoidance mechanism, the robot control and
the complete system architecture. Then, Section 3 presents
the experimental results, and finally, Section 4 concludes the
paper.
System architecture
In general, the overall system is divided into two blocks: the
application block and the object tracking block. The object
tracking block is the core technology of the new version of
SafeBot. This block consists of six main phases: image input,
object tracking, robot control, obstacle detection, obstacle
avoidance, and robot mobility phases. It uses a camera to
learn the movement of tracked objects and then commands
the robot to follow those objects. A range sensor enables the
robot to detect and then to avoid obstacles in real-time while
still tracking the object. If no obstacles are detected, the
462
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
system skips the obstacle avoidance phase and only uses five
phases. Figure 3 shows the entire system architecture. The
following sections explain how each phase works individually,
and how the various phases work in conjunction with each
other in the object tracking block.
Image input phase
The Logitech Web Camera has a fixed view and is attached to
the robotic platform. The camera acquires color-based
320 240 images such that video stream consists of a
sequence of individual images. The camera is tasked to follow
the tracked object.
Object tracking phase
For object tracking, we employ the well known method of Lucas
and Kanade (1981). Although more recent extensions to their
algorithm are available in the literature, we apply Lucas and
Kanades algorithm for computational reasons since we need a
real-time algorithm. We use this method to compute motion
vectors for the tracked object in two consecutive images and
then perform four different directions of the tracked object,
which include moving forward, backward, right, and left.
Because there are so many motion vectors for the tracked object
in the two consecutive images, we use the equation below to
obtain the mass motion vector for the tracked object.
PN
i1 Xi
M 1
N
in this equation, M represents the mass motion vector of the
tracked object (in 1 2 matrix form). The vector Xi represents
each motion vector of the tracked object (in 1 2 matrix form).
The term N represents the total number of motion vectors.
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Figure 3 This block diagram shows the overall system
Robot control phase
Once the object tracking phase is in control of monitoring the
direction of the moving object, it will continue to send the
robot control algorithm the current mass motion vector, M, of
the moving object. It uses this to compute the difference
between current robots location and the origin, M0, which
represents the motion vector of the tracked object that stays in
the center of the image. These operations facilitate the
calculation of the robots control commands. Equation (2)
represents the method used in the robot control phase that
eventually generates the difference vector between the world
coordinate system and the image coordinate system.
" #
Xi Yi
M 2 M 0 W 2
Xj Yj
in equation (2), the bracketed term represents the conversion
matrix which converts the image coordinates into the 2D
world coordinate system. The term W represents the
difference vector in the world coordinate system. Figure 2
shows the concept of the conversion method.
In Figure 4, W0 represents the origin vector, [0, 0], in the
2D world coordinate system. The term R represents the robot
vector, [Rx, Ry], in the 2D world coordinate system. The angle
u describes the angular measure of turn required for the robot
in order to keep the tracked object in the origin of the 2D
world coordinate. The angle u can be computed by dot
product shown in equation (3).
W 0 2 R W 2 R
u cos
1 3 for dynamic goals. Unfortunately, those methods do not
kW 0 2 RkkW 2 Rk
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
After making a turn, the robot then determines the distance to
move, either forward or backward, for keeping it in a fixed
distance from the origin vector, W0.
Obstacle detection phase
Before the robot sends a command to move forward (or
backward), it uses the laser range brick, equipped with the
SICK LMS 200 scanner, to sense if obstacles are in its
projected path. If no obstacle is detected, the robot mobility
phase is activated. Subsequently, the control of the system
returns back to the image input phase. Otherwise, the system
uses the obstacle avoidance phase for generating another
robot control command in order to avoid the obstacle.
Obstacle avoidance phase
The obstacle avoidance phase uses the modified potential
fields methodology mentioned earlier. The advantage of the
potential fields methods is that they are simple algorithmically
and fast computationally. The most significant disadvantage,
however, concerns the local minima problem. Essentially, this
problem means that these methods are not guaranteed to find
a path between the source and destination since a local
minimum can trap the algorithm.
For this application, the robot needs to keep tracking the
object while avoiding obstacles. Since, the object might move
to a new position as the robot is avoiding an obstacle,
traditional potential fields method are not well suited to our
application. because the problem is that most methods
assume the goal position is static, which is not the situation in
our case. Some methods exist in the literature that do allow
generate efficient paths with real-time obstacle avoidance.
463
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Figure 4 Conversion from image to 2D world coordinate system
To deal with this dynamic goal position problem, the obstacle
avoidance algorithm needs to adjust its path corresponding to
the change of its destination during obstacle avoidance. We
propose a new algorithm called the dynamic goal potential
fields (DGPF) method, which is based on the traditional
potential fields method to solve this type of problems. The
DGPF algorithm is based on the following:
1 Using the current configuration, goal configuration, and
sensor data, the robotic platform runs a basic potential
fields algorithm to predict a path.
2 If the goal configuration does not change, then the robot
follows this path to avoid an obstacle.
3 If the goal configuration moves to a new position, which
has a big change from the old position, the algorithm
randomly chooses some points in the predicted path and
runs the basic potential fields method to compute several
paths starting from these points based on current sensor
data.
4 The path with the lowest cost is selected (based on
Euclidian distance). The robot is now uses the new path
to move to the new goal configuration.
5 Repeat it until reaching the goal.
Figure 5 shows the concept of the DGPF method. If the goal
configuration does not change too much during the obstacle
avoidance procedure, the DGPF method is similar to the
traditional potential fields method except that a predicted
path is retained in each step. This predicted path might not be
the same as the exact path taken by the robot because it is
based only on the current sensor data. If the goal
configuration has a large change, the DGPF method has the
capability to quickly adjust its path to move to the new
position with low computational cost.
Like the potential fields method, the DGPF method is a
local path planning method. It does not guarantee an optimal
path to move towards the goal, but it does provide a useful
path for moving towards the goal based on current
information about the environment. Furthermore, to
achieve high computation efficiency, when the goal position
changes to a new position, the DGPF algorithm does not
compute every possible adjusted path, but instead, it
randomly samples several steps along the predicted path.
464
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
Basically the more samples it takes along the predicted path,
the better adjusted path the algorithm can create. However,
the computational cost increases with more samples, and thus
may become to high for achieving real-time reaction.
To achieve more computational efficiency, we do not take
local minima into account while computing adjusted paths.
If an adjusted path is a local minimal, we abandon this path
by simply setting the cost of this path to be infinite.
The cost of each adjusted path can be calculated as
equation (4):
C i C si C mi C li 4
where
Ci is the cost of the ith adjusted path.
Csi is the Euclidian distance from the current position to
the starting point on the predicted path.
Cmi is the Euclidian distance from the starting point on the
predicted path to the end of the ith adjusted path. If this
path is in a local minimal, then Cmi would be set to
infinity.
Cli is the Euclidian distance from the end point of the ith
adjusted path to the goal position.
Here, we only take the robots moving distance into account.
A more reasonable measurement may need to consider the
cost of the robot motion as well. For example, the time for a
robot to make a turn sometimes is longer than to move a
certain distance. A small orientation error will cause large
distance errors after a long distance of moving. So we
probably should add a weight for high orientation values.
Another issue addressing the performance of this algorithm
is the step size. The step size decides the resolution of each
generated path. With a small step size, we can sample more
points along the predicted path, which means we have a better
possibility to calculate the adjusted-path. However, it will take
more time for the robot to finish obstacle avoidance. On the
other hand, with a large step size, the robot can quickly finish
obstacle avoidance if the goal does not move too much away
from the old position. Nevertheless, if the target moves
quickly while the robot is avoiding obstacles, a large step size
may cause high computational cost for the robot to adjust its
path. Since, the object can move at different speeds, a fixed
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Figure 5 The concept of dynamic goal potential field method
step size may cause some problems. A better solution is to use
a dynamic step size where we predict the objects moving
speed and decide the step size correspondingly. When the
object is moving slowly, a large step size is chosen to let the
robot avoid the obstacle quickly. Conversely, a relatively small
step size is set to allow the robot to choose a better adjusted
path to move towards a new position when the object is
moving quickly.
Experimental results
Figure 6 shows the entire system, including the camera, range
sensor, and robotic platform. The mobility of the platform is
accomplished through two independent tracks, which are two
mobility bricks. These two tracks are modular and
interchangeable, and each is capable of moving the robot by
itself. Motion can be controlled directly by the control brick
Figure 6 This system picture shows the platform components
465
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
sending motion commands to each mobility brick
independently via an RS232 connection. Figures 7 through
11 show different experimental results.
In Figure 7, the desired object to track is a human subject
with two automobile tires serving as obstacles. The robot
initially moves backward to avoid the first tire, which is very
close to the beginning position of the robot. Without this
backward movement, the robot could strike the obstacle
(while turning for following the object), potentially losing
sight of the tracked object. In positions (1), (3), and (5), since
there is no obstacle sensed in the laser scan range, the system
does not trigger the obstacle avoidance phase. These
experimental results show that the robot can continuously
track the moving object using the DGPF method to guide the
robot to a new position without colliding with an obstacle.
From Figures 8 through 11, the experimental results
represent the system performance with different robot step
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Figure 7 Experimental result
sizes and object speeds. In these experiments, the motion of the
robot is approximately 1 m/s or slower when it is approaching
the obstacles. In Figure 8, we set that the object speed equal to
2 m/s and the robot step size to approximately 12 cm. Our
experimental results show that this algorithm creates a useful
adjusted path to move toward a new object position. In Figure 9,
we set the object speed equal to 2 m/s and the robot step size is
approximately 50 cm. Our experimental results show that this
algorithm could not adjust its path very well to move toward a
new object position. This experimental results show that a large
step size has trouble in dealing with high-speed objects.
Figures 10 and 11 show where we use the dynamic step size
strategy, which means that the robot can adjust its step size
466
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
according to the object speed. These experimental results show
that dynamic step size typically improves the adjusted path
when the object is moving at a relatively slow speed.
Conclusion
This paper represents a new version of the SafeBot platform
that enables real-time obstacle avoidance and simultaneously
tracks a moving object. This core technology is applicable to
different application areas, such as automated inventory
management, industrial palette distribution, and intruder
surveillance. Currently, our system uses two sensors: a visual
camera to sense the movement of a tracked object and a range
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Figure 8 The speed of the object is 2 m/s, and the robot step size is
12 cm
Figure 9 The speed of the object is 2 m/s, and the robot step size is
50 cm
Figure 10 The speed of the object is 2 m/s with a dynamic robot step
size
sensor to avoid obstacles in real-time while continuing to
track the object. This paper also presents a modified potential
fields algorithm, known as the DGPF method, which
combines real-time obstacle avoidance with object tracking.
Experiments show DGPF can fulfill the requirement of real-
time obstacle avoidance when the tracked object moves at
approximately 1.5 m/s while the maximum robot moving
speed is around 1 m/s.
467
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
Figure 11 The speed of the object is 0.5 m/s with a dynamic robot step
size
References
Barraquand, J. and Latombe, J.C. (1991), Robot motion
planning: a distributed representation approach,
International Journal of Robotics Research, Vol. 10 No. 6,
pp. 628-49.
Borenstein, J. and Koren, Y. (1989), Real-time obstacle
avoidance for fast mobile robots, IEEE Transactions on
System, Man, and Cybernetics B, Vol. 19 No. 5, pp. 1179-87.
Graham, J. and Shillcutt, K. (2003), Robot tracking of
human subjects in field environments, Proceedings of the
International Symposium on Artificial Intelligence, Robotics,
and Automation in Space.
Haritaoglu, I., Harwood, D. and Davis, L.S. (2000), W4:
real-time surveillance of people and their activities, IEEE
Transactions on Pattern Analysis and Machine Intelligence,
Vol. 22 No. 8, pp. 809-30.
Kang, S., Koschan, A., Abidi, B. and Abidi, M. (2004),
Video surveillance of high security facility, Proceedings of
the 10th International Conference on Robotics and Remote
Systems for Hazardous Environments, Gainesville, FL,
pp. 530-6.
Koren, Y. and Borenstein, J. (1991), Potential field methods
and their inherent limitations for mobile robot navigation,
Proceedings of the IEEE Conference on Robotics and
Automations, pp. 1398-404.
Latombe, J.C. (1991), Robot Motion Planning, Kluwer
Academic Publishers, Boston, MA.
Lucas, B.D. and Kanade, T. (1981), An iterative image
registration technique with an application to stereo vision,
Proceedings of the International Joint Conference on Artificial
Intelligence, pp. 674-9.
Perez, L.G., Alegre, M.C.G., Ribeiro, A., Guinea, D. and
Canas, J.M. (2004), Perception and tracking of dynamic
objects for optimization of avoidance strategies in
autonomous piloting of vehicles, paper presented at
International Conference Spatial Cognition, pp. 500-19.
Philomin, R., Duraiswami, V. and Davis, L. (2000),
Pedestrian tracking from a moving vehicle, Proceedings
of the IEEE Intelligent Vehicles Symposium, pp. 350-5.
Qian, C., Page, D., Koschan, A. and Abidi, M. (2005),
A Brick-architecture-based mobile under-vehicle
inspection system, Proceedings of SPIE Unmanned Ground
Vehicle Technology VII, Vol. 5804, pp. 182-90.
 Tracking a moving object with real-time obstacle avoidance
Chung-Hao Chen et al.
Schulz, D., Burgard, W., Fox, D. and Cremers, A.B. (2001),
Tracking multiple moving objects with a mobile robot,
Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition, Vol. 1, pp. 371-6.
Further reading
Khatib, O. (1986), Real-time obstacle avoidance for
manipulators and mobile robots, International Journal of
Robotics Research, Vol. 5 No. 1, pp. 90-8.
About the authors
Chung-Hao Chen is currently a PhD student at the University
of Tennessee. Chung-Hao Chen is the corresponding author
and can be contacted at: cchen10@utk.edu
Chang Cheng is currently a PhD student at the University of
Tennessee.
David Page received his BS and MS degrees from Tennessee
Technological University in Electrical Engineering in 1993
and 1995, respectively. After graduation, David served as a
civilian research engineer with the Naval Surface Warfare
Center in Dahlgren, Virginia for two years where he worked
with distributed architectures for future baselines of the Aegis
Combat System. After returning to academia, David
completed his PhD (2003) in Electrical Engineering at the
University of Tennessee in Knoxville where he currently
To purchase reprints of this article please e-mail: reprints@emeraldinsight.com
Or visit our web site for further details: www.emeraldinsight.com/reprints
468
Industrial Robot: An International Journal
Volume 33 Number 6 2006 460 468
serves as a Research Assistant Professor in the Imaging,
Robotics, and Intelligent Systems Laboratory. His current
research interests are in 3D scanning and modeling for
computer vision applications, modular sensors for robotic
vision systems, and 3D shape analysis for object description.
Andreas Koschan received his Diploma (MS) in Computer
Science and his Dr-Ing. (PhD) in Computer Engineering
from the Technical University Berlin, Germany in 1985 and
1991, respectively. Currently he is a Research Associate
Professor in the Department of Electrical and Computer
Engineering at the University of Tennessee, Knoxville. His
work has primarily focused on color image processing and 3D
computer vision including stereo vision and laser range
finding techniques. He is a coauthor of two textbooks on 3D
image processing and he is a member of IS&T and IEEE.
Mongi Abidi, Professor and Associate Department Head in
the Department of Electrical and Computer Engineering,
directs activities in the Imaging, Robotics, and Intelligent
Systems Laboratory. He received his PhD in Electrical
Engineering at The University of Tennessee in 1987, MS in
Electrical Engineering at The University of Tennessee in
1985, and Principal Engineer in Electrical Engineering at the
National Engineering School of Tunis, Tunisia in 1981.
Dr Abidi conducts research in the field of 3D imaging,
specifically in the areas of scene building, scene description,
and data visualization.