Copyright @ IFAC Telematics Applications in Automation

and Robotics, Weingarten, Germany, 2001

A JAVA BASED SYSTEM FOR NAVIGATION AND

TELE-OPERATION OF A MOBILE ROBOT

Piotr Skrzypczynski

Technical University of Poznan,

Department of Control, Robotics, and Computer Science
ul. Piotrowo 3A, PL-60-965 Poznan, Poland

Abstract: This paper describes a system that allows to tele-operate a semi-autonomous

mobile robot through the intranet/Internet network. The software of this system
has been written entirely in Java to achieve high platform-independence. The
user interface runs as a Java applet within Web browsers. The experiments with
the implementation on a Labmate mobile robot are described. Results of these
experiments show the feasibility of Java as a tool for building user-friendly, networked
robotic systems. Copyright© 2001 IFAC

Keywords: Telerobotics , Teleoperation, Mobile robots, Networks, Software tools

1. INTRODUCTION
The Internet is a very fast evolving global network.
In recent years it has become one of major com-
munication networks and information resources
for the society. Currently, a number of Web sites
provide access to remotely controlled devices such
as cameras, but physical interaction with remote
environments through Internet is still seldom pos-
sible. Intelligent robotic systems, such as mobile
robots , connected to the Internet can be physical
agents allowing users to make changes in the re-
mote environments through the network (Taylor
and Dalton, 2000). Robotic systems operated via
Internet/intranet can have simple, intuitive hu-
man interface based on commonly used software
(e.g. graphical WWW browsers). Such robots can
perform cleaning, delivery, and inspection tasks in
hospitals and large offices, they can be also used in
the entertainment e.g. in exhibitions, fairs or mu-
seums (Thrun et al., 1999) and can assist people
in their everyday activities (Roy et al., 2000).
Nevertheless, there are many challenges in using
the Internet or intranet network as the communi-
cation medium for robots, because such computer
networks introduce non-predictable time delays
between the user and the remote site (i.e. robot).
Because of this, a network-controlled robot should
demonstrate certain degree of autonomy, even if it
is tele-operated.
The aim of the described project was to develop an
intranet/ Internet-based control system for a semi-
autonomous mobile robot . The system, named IN-
TERLABNET enables a remote operator to control
a mobile robot, located at the Mobile Robotics
Laboratory of TU Poznan from any computer
connected to the departments intranet, no matter
where this computer is physically located. After
first tests of this system (Skrzypczynski, 2000) it
has been found out that some improvements are
required, especially that the video feedback can
significantly help the remote user . So a second ver-
sion of the INTERLABNET has been accomplished
in July 2000, and it is described in this paper.
2. INTERLABNET ARCHITECTURE
2.1 Hardware
The test-bed (Fig. 1,2) consists of a robot of the
TRC Labmate type equipped with a low-cost laser

309

intranet I Internet Labmate
Fig. 1. InterLabNet structure - first version
scanning range finder, and infra-red proximity
sensors. All the external sensors, as well as the
robot controller, are connected through the serial
interfaces (RS232). The vehicle is controlled by
its on-board PC-compatible industrial computer
being a node of the computer network. The robot
is connected to the network (intranet) via a wire-
less Ethernet link. Alternatively, the robot can be
controlled via radio-modems from a stand-alone
PC , in this case the on-board computer is not
used.
One of the main problems identified during the
experiments with INTERLABNET (see section 3)
was the inability of the operator to see what the
robot is actually doing. Because the robot does
not have an on-board camera, the video feedback
is provided in the form of images from an overhead
camera, fixed to the ceiling of the laboratory room.
This camera is connected to a stand-alone PC
being a node of the network.
Currently, INTERLAsNET is available to the users
of the departments intranet. The Internet users
outside of the department are not allowed to
control the robot mainly for logistic reasons :
firstly, it is not possible to have the robot on-
line for longer time (because of the limitations
of battery power) , and secondly the navigation
system is still too simple to operate unassisted.
Fig. 2. The structure of the improved InterLabNet
system
310
2.2 Java-based software for a mobile robot
The INTERLAsNET software set-up consists of two
kinds of programs. The first one is the server side
program, that actually controls the hardware (the
robot and all the sensors), and the second is the
client side program that provides the graphical
user interface to the remote operators.
The server communicates with the clients through
computer network , namely the TCP /IP intranet .
The standard Web communication technique is
HTTP (Hyper Text Transfer Protocol) which is
often extended by using CGI (Common Gate-
way Interface) scripts. For the purpose of robot
tele-operation, the most important limitation of
the CGI is that the generated HTML pages are
static , and offer no interaction for the operator.
These standard techniques are not suitable to
build a tele-operation system for mobile robots.
On the other hand , the use of techniques which are
highly system- and vendor- dependent can affect
the port ability of the system, and make it inac-
cessible to some users. Consequently, it has been
decided to write all the system in Java - both the
client and the server side applications.
Java is a full-function , object-oriented program-
ming language. Among other features, it pro-
vides safe error and exception handling, automatic
memory management with garbage collector and
good support for code reuse through formal inter-
face definitions . It has been designed especially to
write portable systems which are able to operate
in a distributed, heterogeneous environment over
the Internet . Web browsers execute Java applets
within WWW pages. This portability of Java ap-
plets allows the remote users of the robot to work
on any hardware/software platform on which a
Java-enabled Web browser is available. Java allows
to implement direct connections over TCP /IP net-
work and thus avoids the most important limita-
tion of the HTTP protocol. Moreover, the applets
enable the user interface to be more interactive
as compared to a sequence of static HTML pages
generated by a CGI script.
Unfortunately, the portable nature of Java has an
impact on its performance (Atherton, 1998; Bol-
lella and Gosling, 2000). The portability has been
achieved by introducing the Java Virtual Machine
(JVM), as the intermediate execution layer. For
the sake of platform independence, the JVM en-
capsulates all services that are specific to the kind
of hardware and operating system, including I/O
operations and networking. Although Java was
originally designed for embedded systems, cur-
rently the language itself and the available JVM
implementations do not support all requirements
for such real-time systems as autonomous robots.
This problem concerns such issues as memory
management, scheduling and synchronization of
threads , interrupt handling, and implementation
of device drivers . Hence, one of the goals of the
presented project was to experiment with Java by
using it to the direct control of devices on board
of the mobile robot .
2.3 Server application
In the first version of INTERLABNET the robot
was controlled from a stand-alone PC through
radio modems (Skrzypczynski, 2000). The server
application written in Java was executed under
JDK 1.2 virtual machine and Windows NT 4.0
operating system. In case of the improved version
of the system the aim was to install the server
on the on-board computer of the mobile robot. As
this is an industrial-PC machine with quite limited
power and resources (486/66MHz) there was no
possibility to use the software from the previous
version in a straightforward way. Because of this,
Linux (Red Hat 6.1) is used as operating system
and Sun JDK 1.2.2 as Java run-time environment.
The HTTP server is used only to deliver the
HTML document with the Java-applet to a remote
user, then the Java-based server and client applica-
tions establish communication through the socket
mechanism without involving the Web-server. Af-
ter it is started in JVM, the server application
initialize serial ports and through them the robot
as well as its sensors. Then, the server waits for
the client requests.
The INTERLABNET server consists of four concur-
rent threads. The main thread is responsible for
operations related to clients. For the safety reasons
the users of the systems have been divided into
two groups - operators and guests. Operators
are those users that are registered, when all other
users attempting to contact the system are treated
as guests. The guests can not guide the robot, they
can only see what it is doing. The main thread
manages the users queue, stores IP addresses,
identifies new users and assigns their rights to
a proper group (guests, waiting operators, active
operator). The same thread manages also the ac-
cess rights of logged clients, i.e. it decides who can
control the robot at a given moment, and who only
receives the sensory data from the robot operated
by someone else. If the current operator issues a
command the main thread receives it and passes
to the control thread.
The control thread actually controls the mobile
robot and serves the active operator. This thread
maintains the communication with the robot's
controller, interprets the high-level operator com-
mands received from the main thread and issues
low-level commands for the robot. Because of the
311
varying bandwidth and possibly large time delays
between the operator and the server the direct
control of the vehicle (e.g. by means of a joystick)
is not suitable. The operator of the INTERLAB-
NET guides the mobile robot by specifying a path
consisting of intermediate goal points. This path is
then followed by the robot, by means of it's local
navigation capabilities. The control thread is also
responsible for reading periodically the robot sta-
tus (including the current position and orientation
from odometry) and for exception handling, e.g. in
the case of a failure of the communication with the
robot.
The third thread communicates with sensors of the
mobile robot. It reads the data from the laser scan-
ner and builds the local occupancy grid according
to the Histogramic In-Motion Mapping (HIMM)
algorithm proposed by Borenstein and Koren
(1991) . The local occupancy grid provides the base
for obstacle avoidance, which is implemented by
using the Vector Field Histogram (VFH) method
(Borenstein and Koren, 1991). The robot under
control of INTERLABNET does not have high de-
gree of autonomy due to the limitations of the
on-board sensors (Skrzypczynski, 1997). However,
the basic autonomous behaviours like emergency
stop and simple obstacle avoidance have been im-
plemented in the sensor-processing thread of the
server to ensure the safety of navigation in the face
of time-delays and possible disruption of commu-
nication with the remote operator. All navigation
algorithms have been implemented in Java and
integrated in the server.
The data collected by the server i.e. sensory read-
outs, robot position and status, and current sta-
tus of the server are passed to the main thread
and then broadcasted to the clients in the form
depending on their status/rights. Moreover, the
local occupancy values are also transferred to the
remote users, and then used to build global occu-
pancy map in the clients applet.
The last thread of the server is the connection
monitor. It's task is to wait in a loop for the
connection requests from new clients. If a new
client is connected, this thread passes his address
to the main thread, and continues to wait for
next connection. This separate thread for new
connections helps to avoid delays and thus makes
the whole system more accessible.
2.4 Video feedback
Where other Web-based mobile robots (e.g. Xavier
(Simmons, 1998) , Minerva (Thrun et al. , 1999))
provide video feedback from on-board cameras,
the INTERLABNET takes a different approach
and allows the remote users to see the robot
 camera hnages(Java)
operator / guest applet (Java)
login
(JavaScript)
welcome page (HTML)
Fig. 3. The configuration of InterLabNet Web site
from a "bird's eye" perspective through a ceiling-
mounted camera. Currently, only one camera is
used , but thanks to the fish-eye lenses an image
from this camera covers almost entire floor of the
lab. The B/ W CCD camera sends the video signal
to a frame grabber placed in a stand-alone PC ,
which runs under Windows 9x. The video frames
are captured by a separate GetCam program (be-
ing the only part of the system not implemented in
Java), then converted to JPEG format and stored
in a file . The HTTP server (Apache 1.3.6) sends
these images to client applets.
The PC hosting the camera is also a Web server
for the introduction WWW pages. These " wel-
come" pages provide a short presentation of the
INTERLABNET concept, and the login service (im-
plemented in JavaScript) for users . Clients logged
in are directed via hyperlinks to the actual Web
server of the system , hosted on board of the robot.
If the on-board server can not be reached , the
user still can see images from the overhead camera
available through a separate applet (Fig. 3).
2.5 Web-based operator interface
The client-side applet is down loaded from the
Web-server , and runs within a Java-enabled browser
(e.g. Netscape Navigator) . It establishes a con-
nection with the server application via a socket
and enables the user to interact with the robot.
There are separate applets available to the guest
and to the registered operators. Which one is sent
to the client is decided by the login service. The
operator's applet gives him the control over the
vehicle , while the guest's applet lets him only to
watch the operation of the robot .
312
InterLabNet
server
(Java)
mobile robot
The operator can specify a path for the robot
by clicking in the simplified map of the robot 's
work area (Fig. 4). The map can display the
2D predefined model of the environment , current
robot position and current path (lower-left part
of the window) . The operator can also change the
velocity and acceleration of the robot , stop it or
reset, by using the" dashboard" in the central part
of the applet 's screen . The client's applet features
also the status-message monitor (upper-left part
of the window). This monitor allows the operator
to watch the communication between the applet
and the server, as well as the current status of the
server, and robot-server communication.
The upper-right part of the applet contains the
camera feedback window. The view is updated
periodically, with the delay time chosen by the
user from the options available below the image
window. Currently, the minimal delay is 1 second ,
what does not ensure a " live" video feedback. The
image refresh rate is limited mainly by the frame
grabber used , and the current implementation of
the GetCam capturing program, and it is expected
that this parameter can be improved .
Because the human operator does not have di-
rect video feedback from a robot-mounted cam-
era, the graphical presentation of the synthetic
environment model becomes very important. The
sensor-based map displayed to the remote oper-
ator (lower-right part of the window) is an oc-
cupancy grid. The grid-based map has been cho-
sen because such map tolerates sensory data un-
certainty and ambiguity so it can serve also in
weakly-structured environments. In the first ap-
proach a slightly modified version of the algo-
rithm proposed by Elfes (1990) has been used
to integrate the range measurements into a map

[~~~~~~
.~------'i ::t=J--1~ :;
:::f:Ij~ .
"'"-__ .....
.,-.,- r
I could not re-connect to the robot , after the robot 's
L ...-)(ro-_'f~'.
... -ro::===- ' ~
~. ,.~, ,M'>.<. ....~_...,< I:
Fig. 4. The operator 's applet graphical interface
(Skrzypczyriski , 2000). This algorithm is compu-
tational intensive and its implementation in Java
on the on-board computer can rise problems . In
contrary, the HIMM method does not involve the
use of probability functions, it only increments and
decrements cells in the map for each range reading ,
what requires minimal computational overhead.
Moreover , the fast map building allows the robot
to use the resulting local map for obstacle avoid-
ance by means of the VFH method. The original
HIMM algorithm has been modified in order to use
laser scanner data instead of ultrasonic rangefinder
readouts , and implemented in Java. Because the
scanner has a narrow beam and the readouts do
not suffer from specular reflections and crosstalks,
it was possible to simplify the data integration
method by avoiding the filtration operators, used
by Borenstein and Koren (1991) to compensate for
the effects caused by in-motion use of sonars.
The local occupancy map is produced by the
server application, and then sent to the clients.
The client's applet receives the data and integrates
them into a global occupancy grid. Running the
global occupancy map building process on the
local machines helps to avoid large communication
gaps , that can arise due to the transmission of
huge data packages (like the global map) to clients.
3. EXPERIMENTAL VALIDATION
To validate the approach and the robustness of the
software a series of experiments has been carried
out. With the first version of INTERLABNET the
robot has been controlled by using several different
computers (PC and SUN) working within the
department intranet. During these experiments
the robot was guided also by untrained operators,
for whom working with INTERLABNET was a first
contact with mobile robots.
From these experiments it has been found out
that the reliability of the Java-based software is
good , despite of some technical problems with the
robot. The Java application (server) running III
JVM under Windows NT on a stand-alone PC
(Pentium/233MHz) was fast enough to control the
mobile robot and its sensors via radio modems.
The communication between the server and the
robot was stable even when the number of simul-
. ___.ff I ~ 7 r)r , r ,e taneously connected clients was high. The problem
......,...... -,.,.... was the inability of the server to recognize the
. P' " -
failures of the robot hardware , for instance , it
controller " reset" due to the low battery power .
Similar experiments have been carried out with
the second version of Il'.'TERLABNET. The results
obtained were better because of the improved
server functionality (scheduling of users requests ,
new obstacle avoidance functions , more robust
protocol for communication with the robot), and
the visual feedback available to the remote oper-
ator. The possibility to visually track the robot
on the camera window provides a better feeling of
reality, even with the limited image refresh rate.
The system has been tested in two hardware con-
figurations, namely the target configuration of the
new version (Fig. 2) , and the configuration with
the INTERLABNET server on a stand-alone PC
running under Windows NT, and radio-modem
connection to the robot (Fig. 1). This was pos-
sible without any modification of the software ,
because of the portability of the server written
in pure Java. The location of the server appli-
cation on the robot's PC improves the safety
of the whole system, because the communication
between the server and the robot's controller is
" local" (through cables), in contrary to the radio-
modem communication.
Although the system is completely based on Inter-
net technologies, the server has enough autonomy
to compensate the effects of possibly unreliable
wireless communication. Even if the communica-
tion is lost completely, the robot protects itself
from collisions with obstacles. The problem is the
limited power of the on-board computer in the cur-
rent experimental set-up. It affects mostly the low-
level algorithms, like map building and obstacle
avoidance, and subsequently limits the safe speed
of the vehicle to about 0.2 m/so It limits also the
number of clients which can connect simultane-
ously.
4. CONCLUSIONS
The contribution of this work is the implementa-
tion of the entire mobile robot navigation system
in Java. This system proved to be capable of al-
lowing safe navigation in a structured laboratory
environment and execution of tasks issued by sev-
eral remote users.
313
The lesson learned since the first version became
operational (June 1999) is that the autonomy is
a key feature for a networked mobile robot, even
if it is largely tele-operated, and that the robot
must have enough "intelligence" to tolerate un-
predictable gaps in communication or even mis-
leading commands from non-expert users. On the
other hand, it has been found out that Internet
communication techniques are enough flexible and
reliable to be used for tele-operation of mobile
robots in real , indoor applications. Finally, the
feasibility of Java language and Java run-time en-
vironments as the base for robotic applications has
been tested. The use of Java has many advantages,
like portability and built-in networking support,
but the possibility of implementing real-time pro-
grams is somewhat limited . It is expected that
the introduction of the new real-time specification
for Java (Bollella and Gosling, 2000) will help to
overcome these limitations.
Mobile Robot via Internet. Proc. Int. Conf.
Mechatronics 2000, Warsaw , 380-384.
Taylor, K. and B. Dalton (2000) . Internet Robots:
A New Robotics Niche. IEEE Robotics and
Automation Magazine 7(1) , 27-34.
Thrun, S., M. Bennewitz , W . Burgard , A. Cre-
mers, F . Dellaert, D. Fox, D. Hiihnel , C.
Rosenberg, N. Roy, J. Schulte and D. Schulz
(1999). MINERVA : A Second-Generation Mu-
seum Tour-Guide Robot. Proc. IEEE Int .
Conf. on Robotics and Automation, De-
troit , 1999-2005.

5. ACKNOWLEDGMENTS

Thanks are due to P. Grajdek and D. Jopek for

their contribution to the implementation of the
described system.

6. REFERENCES
Atherton , R. (1998) . Moving Java to the Factory.
IEEE Spectrum 35(12) , 18-23.
Bollella, G. and J . Gosling (2000) . The Real-
Time Specification for Java. IEEE Computer
33(6) , 47-55 .
Borenstein, J. and Y. Koren (1991). Histogramic
In-Motion Mapping for Mobile Robot Obsta-
cle Avoidance. IEEE Journal of Robotics and
Automation 7 (4), 535- 539.
Elfes, A. (1990). Sonar- Based Real- World Map-
ping and Navigation. In: Autonomous Robot
Vehicles ( I. J. Cox and G. T . Wilfong Ed.),
pp. 233- 249, Springer- Verlag , Berlin.
Roy, N. , G. Baltus, D. Fox, F. Gemperle, J . Goetz,
T . Hirsch, D. Magaritis, M. Montemerlo, J .
Pineau, J. Schulte, and S. Thrun. (2000). To-
wards Personal Service Robots for the El-
derly. In: Proc. of the Workshop on Interactive
Robotics and Entertainment (I. Horswill and
T . Balch, Ed.) CMU, Pittsburgh.
Simmons, R. (1998). Xavier: An Autonomous Mo-
bile Robot on the Web. Proc. IROS'98 Work-
shop on Robots on the Web, Victoria, 43-48.
Skrzypczynski, P. (1997) . Supervision and Tele-
operation System for an Autonomous Mobile
Robot. Proc. Int. Conf. on Intelligent Robots
and Systems, Grenoble, 1177-118l.
Skrzypczynski, P. (2000). InterLabNet: A Java-
based System for Remote Operation of a

314