Development of Haptic Virtual Reality

Environment for Planning Micromanipulations

T.S.Simil
Assembly and Testing Division, Spacecraft Mechanisms Group, ISRO Satellite Centre, Bangalore, India
(Email: simil@isac.gov.in)

Abstract: Haptic technology augments the level of human-
computer interaction by incorporating a new display interface in
the form of tactile sensation. Haptics finds one of its potential
engineering applications in the regime of microassmbly. This
paper presents the development of a software tool which has been
realized for planning micromanipulations in a microassembly
domain with the aid of a haptic virtual reality environment. This
development was realized through haptic simulations and a
graphic environment representing the microassembly domain.
An assistive technique to guide the user during
micromanipulation with the aid of haptic interface has been
proposed.
I.INTRODUCTION
Haptics is a relatively new field of interface technology which
augments the human-computer interaction by incorporating a
display technology through which virtual objects can be
physically palpated. Integrating force feedback with virtual
reality (VR) simulations provides the user the feel of tactile
sensation on virtual objects in the form of object stiffness,
surface contact geometry and friction [1].
Haptics finds one of its potential engineering applications in
the regime of microtechnology. In microassembly, parts in
the size of few microns are needed to be manipulated and
assembled with sub-micron precision. Uncertainties in the
microworld such as dominance of adhesive forces over
gravity, meager or no force feedback, noisy visual information
etc pose challenges to the microassembly and associated
micromanipulation tasks. Suitable supportive technology to
facilitate proper planning and training in executing the micro
assembly tasks is essential to handle these issues. Use of
haptic technology has proved beneficial into such applications
This paper presents the development of a haptic virtual
reality environment which can aid in generating strategies for
micromanipulation planning and guidance through
simulations. A brief introduction on haptics is presented in
section II. Scope of virtual simulations for micro assembly
and related research is discussed in section III. The
development of haptic simulation environment and the
description of test cases are described in sections VI and V.
II. HAPTIC INTERFACE TECHNOLOGY
Haptic technology adds a new dimension into the human-
computer interaction domain by incorporating the sense of
touch. User is made to ‘feel’ the virtual objects through the
haptic interface device in the form of object stiffness, surface
contact geometry, and friction [2]. This is accomplished by
integrating the force feedback with the VR simulation as
shown in fig 1. Whenever an interaction occurs between the
user and the virtual objects, feedback forces are generated
through the haptic device. The haptic rendering engine checks
for the user – virtual object collisions and determines the
magnitude and direction of force to be returned. Simulation
algorithms integrate the haptic and graphic rendering engines
and provide a suitable output to user based on the program
logic.
III. SCOPE OF HAPTIC SIMULATION FOR MICROASSEMBLY
Manipulation of parts in a microassembly domain is carried
out through sub millimeter sized micro manipulators. Typical
micro assembly tasks need sub-micron precision and hence a
good user interaction with the assembly system is a must.
Generally the feedback to the user is in the form of a
magnified visual display through a microscopic camera
system. User has to perform the 3D operations with the aid of
this 2D visual display.
Fig 1: Virtual environment incorporating haptic and visual feedback. Haptic
interface device provides bi-directional flow of information between the user
and the virtual environment.

1
 The forces in the micro level are very small in magnitude as
compared to gravity based and other interactive forces in
macro level and can be considered negligible to be felt by the
user in the absence of a force magnification system. This
results in a lack of hand-eye coordination during the micro
assembly tasks. Moreover the visual information is
constrained by the working range of microscopes and there is
limited viewing flexibility.
Another major challenge in micromanipulation is the
dominance of adhesive forces. An adhesive force comprise
three major components namely the van der Waals,
electrostatic and surface tension. In the micro level, these Fig. 3: Test setup showing the desktop PC interfaced with Phantom Omni
forces dominate over gravity and make the micro level haptic device.
operations complex. Fig 2 shows the well known problem of
sticking of micro parts onto the gripper [3]. 1) Building micromanipulation scene: User can select
the required micro objects, substrates and
A. Related Work
micromanipulator from an openGL model library to
Research carried out in this field generally focus at the tele-
build the microassembly scene. User can arrange the
operated environments coupled with haptic VR simulations.
models at any required position / orientation in the
An automatic micro assembly system assisted by virtual
scene.
simulation can be found in [4]. A micro manipulation system
2) Manipulation: User can activate and use the virtual
through tele-interface with haptic assistance is presented in
manipulators through the haptic device to perform
[5]. The development of a virtual tool kit with visual and
various micromanipulation exercises by picking up or
haptic feedback to test manipulation strategies at nano-scale is
pushing the micro parts. Once the required
described in [6]. A 3D micro world developed with haptic
manipulator is activated, the haptic device stylus
feedback for simulating the effects of van der Waals and
movements will get coupled with the manipulator
electrostatic forces is presented in [7].
model.
IV. DEVELOPMENT OF HAPTIC SIMULATION ENVIRONMENT
3) Manipulation guidance:. A guidance system which
The haptic micromanipulation environment has been designed can physically guide the user’s hand movements
and implemented at the Real time collaborative environment through a pre-defined path has been provided in the
laboratory of IIT-Kharagpur, India. The development was environment. User is made to feel an attractive force
carried out on a Pentium -IV desktop PC running on Linux towards the path geometry so that he can make
Fedora Core-4. The haptic virtual simulation was realized deviation free manipulations through this path.
using C++ and the software libraries OpenGL and 4) Data recording: This feature enables a real time
OpenHatpics [8]. The haptic interface device used was capture of positional and force feedback information
Phantom Omni from Sensable Technologies [9]. Fig 3 shows during simulation. This data can be used for
the set up used for development. replicating the manipulation exercises automatically
The following features are provided in the simulation using a robot manipulator arm or similar device.
environment to enable the user to perform and plan the
micromanipulation The development was carried out with modular software
architecture to provide the above features. Following sections
describe the various modules designed for the same.

A. OpenGL Model Library

A library of virtual models was created using openGl. The
models in the library represent the micro parts for assembly,
micro-manipulators and substrates. Fig 4 shows the
architecture of the model library and how the user can build
the manipulation scene using the same.
Fig 2: Effect of adhesive forces during micromanipulation. The micro part
sticks to the manipulator during approach & release and make the task
complex

2
 Fig 5: The haptic device movement are captured at every instant and mapped
to the virtual scene and fed into the manipulator animation module. This
results in every movement made on the device stylus to suitably replicate into
the manipulator model movements through animations.

Fig 4: The model library enables the user to select the appropriate models and
assign the material properties & transformation required for building the
manipulation scene
Each model is built with a standard size and associated
with default position and placement in the manipulation scene.
This can be modified by the user according to the application
needs. The required material can be selected and assigned to
each model. The properties of the material are stored in the
material data file. Each material is associated with the
properties such as stiffness, static & dynamic frictional
coefficients and damping coefficient for the haptic interaction
and the Hamakers constant values for the adhesive force (van
der Waals) computation [10].
B. Manipulator Simulation and Control
An algorithm in this module couples the manipulator
geometry transformations with the stylus movements of the
haptic device. This is accomplished by capturing the required
position and orientation of haptic device stylus and suitably
transforming it to update the geometry translation and
rotation. By this way user can have a real time control of the
virtual manipulator through the haptic device. User can define
up to 6 degrees of freedom for the virtual manipulator. Fig 5
illustrates the methodology of manipulator simulation.
D. Force Simulation
In the micro level, adhesive forces (van de Waals, electrostatic
and capillary forces) are predominant over the gravitational
and inertia forces. Among the adhesive forces the capillary
force can be eliminated with proper humidity control.
Similarly the use of static charge controlled clean room can
substantially reduce the effect of electrostatic forces. For the
present simulations it is assumed that these two forces are well
controlled and only van der Waals force exist between the
objects. The van der Waals force is originated due to the
momentary dipole moment between atoms resulting from
interaction between electrons rotating in the outermost bands
around the nucleus. In general, the van der Waals force
modeling between two surfaces is based on the object material
characteristics and the surface geometry as shown in fig 6.
The governing equations for van der Waals force modeling are
as follows.
The van der Waals force Fvdw between two spheres of radius
R1 and R2
/12 (1)
Where, √ 1 2 (2)

C. Collision Detection and Control

The collisions between the geometric models in the
environment are controlled by this module. Axis Aligned
Bounding Box (AABB) technique [11] is adopted for
determining the collisions. The focus here is only to avoid
collision between the geometries. Once the collision is
detected between the geometries, the algorithms arrest the
movements of the objects towards the direction of collision.
Fig 6: Notations for van der Waals force determination.

3
Between a spherical particle of radius R and a flat surface
/6
Between two plane surfaces
/ /6
Where, Ac is the contact area and A is the Hamaker constant
which is a function of object material.[12].
The gravitational force is influenced by the mass of the
object and is generally negligible in the micro domain.
ρg
Where, V is the volume of the object, is the density and g
is the acceleration due to gravity.
The force computation module determines the van der
Waals forces experienced between the micro objects, between
micro object and manipulator and between the micro object
and the substrate in real time. The net force effect on the
object dynamics during manipulation is analyzed by the force
simulation module and proper haptic and graphic updates will
be determined by the object dynamic simulation module.
Graphic updates are in the form of object movements based on
the net interaction forces (Eg. Adhesion force between the
manipulator and object as well as between the substrate and
object when manipulator touches the object). Haptic updates
gives the magnified force effect to user when object jumps and
sticks to manipulator Fig 7 shows the module architecture.
Fig 7: Force simulation module determines the net adhesive force effect
between objects and generates the appropriate graphic and haptic rendering.
E. Manipulation Guidance System
A novel manipulation guidance system is designed in a
(3) view to assist the user in making straight line motions which
are most often compatible with straight line motions creatable
by table / manipulator tip movements that are available in
most of the microassembly systems. In the current work
(4) interpolated motions are not being considered to keep errors of
interpolation out of consideration.
The manipulation guidance system is provided with a path
designer module which facilitates the user to generate the
manipulation trajectory by inputting the coordinates of the
final positions to which the parts to be manipulated. The
algorithms in the module generate the path with straight line
segments from the current position of the parts to the final
location. (Point to point motion planning with intermediate
(5) waypoints using linear interpolation). A graphical
representation of the path will be displayed on the
manipulation scene.
The openHaptics API facilitates a feature to constrain the
haptic interface point (HIP) [2] position to collocate with a
particular geometry. This results in the self movement haptic
device stylus to the geometry and remains to stay in a state of
collision unless an opposite force is applied by the user. In
effect user feels an attractive force towards the geometry. This
feature is exploited to generate the physically assisted
manipulation guidance technique. As soon as the user touches
the object, the algorithms enable the above described feature
to constrain the HIP with the manipulation path geometry.
User experiences the attractive force towards the path
geometry and will be guided through ittowards the final
location. Fig 8 illustrates the software architecture and Fig 9
shows the environment with the manipulation trajectory.
Fig 8: The manipulation guidance system generates the manipulation
trajectory and guide the user hand movements along it by creating a virtual
attractive force.
4

Fig 9: Simulation environment showing beginning and intermediate stages of
pick and place manipulation using virtual adhesive gripper and with the aid of
manipulation guidance system.
F. Haptic Menu Interface
A novel haptic user interface system has been created in which
the menus can be operated through the haptic device. This
feature avoids the practical difficulty of operating the haptic
device and the mouse / keyboard simultaneously so that user
can concentrate mainly on the haptic device. This improves
the level of user immersion in to the virtual domain. The
menu system consists of 3D buttons and sliders which the user
can operate as if in the real world. This also consists of
graphical user menus for the non-frequently used functions.
The information related to force feedback, manipulator
positions, adhesive forces etc are displayed in the
environment. Fig.10 shows the user interface system.
Fig 10: The Haptic menus provided in the simulation scene facilitates the user
to operate it through haptic device and there by improves the user immersion
to the virtual environment
5
G. Data Capture and Post Processing
During simulation the positional information of models in the
scene, the calculated adhesive and gravitational forces and the
haptic feedback forces are captured in real time. This data
was processed using matlab to analyze the user performance at
each trial. Fig 11 shows this module architecture.
H. Master Simulation Control
The above modules are integrated together to form the
simulation environment through the master simulation control
module. The simulator is designed in such a manner that it
may easily be integrated to an experimental
micromanipulation environment which may be developed and
interfaced later for practical verification. In this paper
however, the simulator and planning of motions in the virtual
environment is addressed for experiments as detailed in the
next section.
V. TEST CASES
To demonstrate and evaluate the performance and capabilities
of the development the following three virtual manipulation
experiments have been considered. 1) Manipulation and
assembly of a micro gear into a shaft using a mechanical
gripper 2) Pick and place manipulation of rectangular blocks
using an adhesion based gripper and 3) Manipulation of
nanoparticles by controlled pushing. The cases 1 and 2 were
performed with the aid of manipulation guidance system.
A. Manipulation and Assembly using Mechanical Gripper
Manipulation of a micro gear model to assemble with a micro
shaft model has been simulated in this case. The openGl
models were sized with internal diameter of gear and external
diameter of shaft as 10µm. The simulation domain was
created with dimension of 300 x 200 x200 µm.
Fig 11: The positional and force data is captured in real time during
simulations and processed to evaluate the user performance
 The mechanical gripper (which is the manipulator in this
case) was modeled with two jaws which can move opposite to
each other to hold the object. The gripper was simulated with
4 degrees of freedom (DOF) which comprises the translation
in X,Y and Z directions and the rotation about the gripper
axis. The movements of the haptic device stylus were coupled
with the animation algorithms of the gripper model as
explained in the previous section (IV-B). A virtual switch in
the form of hapic menu was provided in the scene to enable
the user to initiate the manipulator control through the haptic
device. The jaw movement simulations were designed to
control through the push buttons provided in the haptic device.
A holding fixture with two jaws was modeled and
simulated. The jaw movement animation of this model was
controlled through the keyboard. The magnification factor
for display is chosen such a way that one pixel represents
0.25µm. The force magnification factor is selected as 105.
Fig 12 shows the simulation environment.
Fig 12: Sequences in virtual micro assembly. 1) The scene before
commencing the manipulation. 2) The shaft is being manipulated toward the
holding fixture 3) The gear is being manipulated towards the shaft which is
held by the holding fixture. 4) The gear is being assembled into the shaft with
the aid of manipulation guidance system.
6
The parts were initially positioned in the scene in a random
fashion. The simulation commenced with gripping the shaft
using the gripper. The ‘sticking problem’ due to adhesive
force was seen during the gripping process. The shaft was
then manipulated towards the holding fixture and made to
hold with the fixture jaws. Subsequently the gripper jaws were
released. During the gripping simulations, the output from
collision detection module was used to stop the jaw movement
animations. The gear was also picked up in a similar manner
and manipulated towards the shaft for assembly. The
manipulation guidance was activated at this juncture to aid the
user to locate the shaft axis and manipulate the gear along
with it. During the assembly, the assembly forces were felt to
the user in a magnified scale.
A force range constraint was defined for performing the
assembly. User was made to apply a threshold force to insert
the gear into the shaft. If the force tends to exceed a value
that can damage or deform the parts, user is alerted with an
opposing force effect in the form of a jerk and a graphical
indication by changing the path geometry color.
It was observed that initially the user found it difficult to
grip the objects due to the adhesive force simulation. The
gripping could not be carried out if the object sticks on the
outer part of the jaw. But with repeated trials user has
succeeded in accomplishing error free manipulations.
B. Pick and Place Manipulation using Adhesive Gripper
The pick and place simulation was carried out by modeling an
adhesion based micromanipulator. The adhesion based
manipulator makes use of adhesive forces for the gripping and
release. For gripping the adhesive forces between the gripper
and the object must be greater than the sum of adhesive forces
between the object and substrate and the gravitational forces.
Whereas during release of the object, this condition has to be
reversed. One of the techniques to accomplish this is to
decrease the contact area between the manipulator and the
object by rotating the manipulator and thus reduce the van der
Waals forces [13]. Fig 13 shows the gripping technique used
by the adhesive gripper.
The adhesive force computation algorithms were modified
for this experiment to determine the change in contact area
due to gripper rotation. Considering the size of the parts and
the possible deformations that occur during contact, it was
decided to approximate the change in contact area as a linear
function of change in angle.
Fig 13: The adhesive gripper make use of the adhesion forces for both
gripping and release by rotating the gripper and thereby changing the contact
area with the object.
 Three rectangular blocks were modeled as objects with
sizes (20 x20 x10),(10 x 10 x 10) and (5 x5 x10) micron. The
magnification factor for display is chosen such a way that one
pixel represents 0.25µm. The force magnification factor is
selected as 105.
The micromanipulator was modeled with 5 DOF
comprising translation along X,Y,Z axes and rotation about
the roll and pitch axes. The van der Waals force has been
modeled based on the Hamaker constant of the materials [11].
The object material was assumed as copper and the
manipulator as gold coated glass with Hamaker constant value
as 40 x 10-20J. The substrate material was assumed as
polystyrene with Hamaker constant 8 x 10-20J. Fig. 14 shows
the simulation environment.
The objects were initially positioned and oriented in a
random fashion. The final position and orientation of the
objects were predefined. The trajectory through which the
objects are to be manipulated was generated by the
manipulation guidance system as explained in section IV-E.
The objects were picked up by the gripper using the simulated
adhesion forces. As soon as the object is gripped, the
manipulation guidance got initiated and the manipulation
trajectory was displayed in the scene. The objects were
manipulated towards the final positions through the trajectory
aided by the force guidance. The objects were properly
oriented and released at the target position.
Fig 14: Stages of virtual pick and place manipulation with the aid of
manipulation guidance. 1) The scene before commencing the manipulation 2)
The first object is being picked up using adhesive force 3) The second object
is being placed on the target point 4) The end of manipulation.
7
It was found that the use of manipulation guidance is very
effective in accomplishing error free manipulations. The
difficulty was found only in orienting the objects to the
desired fashion at the final release. A guidance methodology
to control the manipulator rotations would be useful to counter
this problem.
C. Manipulation by Controlled Pushing
An attempt was made to simulate a nanomanipulation. A
virtual nano environment has been created with virtual models
of an Atomic force microscope (AFM) manipulator, nano
particles and substrate [14]. The AFM manipulator is
provided with 4 degrees of freedom comprises translation
along X,Y,Z axes and rotation about the manipulator axis.
The substrate was modeled with an assumed surface topology
as shown in fig 15. User was initially presented with
randomly positioned virtual nano particles on the substrate.
The virtual nano particles were manipulated to the desired
locations using the manipulator by controlled pushing. It is
assumed that the adhesive forces between the substrate and the
nano particles are more than that of between the manipulator
and particles. The nano level forces during manipulation
were presented to the user in a magnified scale.
Fig 15: Stages of virtual manipulation through controlled pushing. 1) The
scene before commencing the manipulation 2& 3) Particles are being
manipulated towards the target locations 4) The end of manipulation.
 VI CONCLUSION

Haptic simulations can be effectively applied in developing

supportive technologies for the microassembly and associated
micromanipulation problems. The presence of predominant
adhesion and related surface forces pose challenge to the
planning and design of strategies for the micromanipulation.
This paper addresses the guidance and manipulation in such
environments using a haptic interaction environment and
discusses the details of such a software tool that has been
developed and demonstrated for some simple micro
manipulation examples.
ACKNOWLEDGMENT
The author wish to thank Dr. C.S. Kumar, Associate
Professor, Department of Mechanical engineering, Indian
Institute of Technology, Kharagpur, for his guidance and
support towards the successful completion of this project.
REFERENCES
1. Burdea, “Force and Touch Feedback for Virtual Reality”, John Wiley &
Sons, USA, 1996.
2. Kenneth Salisbury, Francois Conti and Federico Barbagli “Haptic
rendering: Introductory concepts”, IEEE Computer Graphics and
Applications, Jan 2004, pp. 24-32.
3. Ronald S Fearing, “Survey of sticking effects for micro parts handling”,
Proc. IEEE International conference on Robotics and Intelligent systems,
Pittsburg, Aug 1995.
4. A.Ferreira, C.Cassier and S.Hirai, “Automatic Microassembly system
Assisted by Vision Servoing and Virtual Reality” IEEE/ASME transactions
on mechatronics, Vol 9, June 2004.
5. Mehdi Ammi, A.Ferreira and J.G.Fontanie “Virtulized Reality Interface for
Tele-Micromanipulation”, Proc. IEEE International conference on Robotics
& Automation, New orlands, 2004.
6.Aydin Varol, Ihsan Gunev and Cagatay Basdogan, “A virtual reality tool kit
for path planning and manipulation at nano scale” Proc. Symposium on haptic
interface for virtual environment and teleoperator systems, 2006.
7. C.J van Strijp, H.H. Lanjen, M. Onosato. “The Application of Haptic
Interface on Micro assembly”, Proc. Symposium on haptic interfaces for
virtual environment and teleoperator systems , 2006.
8. Brandon Itkowitz, Josh Handley and Weihang Zhu, "The OpenHaptics™
Toolkit: A Library for Adding 3D Touch™ Navigation and Haptics to
Graphics Applications," Proc. First Joint Eurohaptics Conference and
Symposium on Haptic Interfaces for Virtual Environment and Teleoperator
Systems (WHC'05), 2005
9. Massie and Salisbury,”The PHANTOM Haptic Interface: A Device for
Probing Virtual Objects”, Proc. IEEE International conference on Robotics
and Automation, Minopolis, 1996.
10. N.Israelachvilli, “The Nature of van der Waals Forces”, Contemporary
Physics, Vol. 15, No.2, pp.159-177. 1974.
11. A.Garica Alonso, Nicolas Serrano and Juan Flaquer, “ Solving the
collision detection problem”, IEEE Computer Graphics and Applications,
13(3):36-43,1994.
12.Hamaker.H.C, "The London-van der Waals: Attraction between spherical
particles," PhysicaIV, No. 10, 1058, 1937.
13.John. T. Feddema, Patrick Xavier and Russell Brown, “Assembly planning
at the micro scale”, Workshop on precision manipulation at the micro and
nano scales” Proc. IEEE international conference on Robotics and
Automation, Leuven , 1998.
14.Guangyong Li, Ning Xi, Mengmeng Yu and Wai Keung Fung, “3D
Nanomanipulation Using Atomic Force Microscopy”, Proc. IEEE
International conference on Robotics & Automation, Taipai,Taiwan, 2003.

8
9