IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 15, NO.
6, DECEMBER 2010
ViKY Robotic Scope Holder: Initial Clinical
Experience and Preliminary Results Using
Instrument Tracking
Sandrine Voros, Georges-Pascal Haber, Jean-François Menudet, Jean-Alexandre Long, and Philippe Cinquin
Abstract—Robotic endoscope holders constitute an alternative
to complete telesurgery systems, by offering a “third hand” to
the surgeon during a laparoscopic procedure. ViKY robotic scope
holder (Endocontrol, Grenoble, France) is a lightweight, steriliz-
able body-mounted robot with 3 DOF. In this paper, we present the
specifics of this robot, the newly developed XL version dedicated
to single-port surgery and the initial clinical experience. We have
proposed a method to control the robot, based on the detection and
tracking of surgical instrument from image analysis and shape pri-
ors, to enrich the interaction between the surgeon and the robot.
We present here, our work in progress toward its integration into a
probabilistic framework, with the aim of improving the method’s
speed and robustness. We also present the surgeons’ viewpoints on
the feasibility of its integration into the operating theater.
Index Terms—Automatic instruments detection and track-
ing, clinical experiments, laparoscopy, robotic endoscope holder,
robotic surgery, vision-based control.
I. INTRODUCTION
APAROSCOPIC surgery, also called keyhole surgery, uses
L tiny incisions, usually less than 1 cm to perform intraab-
dominal or intrathoracic procedures. This approach offers de-
creased blood loss and postoperative pain, in addition to shorter
hospital stay and convalescence, while offering better cosme-
sis [1]. This technique requires the use of a laparoscope and spe-
cific instruments designed for this approach. While the surgeon
Manuscript received March 1, 2010; revised July 3, 2010; accepted Au-
gust 30, 2010. Date of publication November 15, 2010; date of current version
December 15, 2010. Recommended by Guest Editor R. L. Galloway. This work
was supported by the French National Research Agency (ANR) through its
TecSan Program under Project ROSACE ANR-06-TecSan-008.
S. Voros is with the Techniques de l’Ingénierie Médicale et de la Complexité-
Informatique, Mathématiques et Applications de Grenoble Laboratory, Cen-
tre National de la Recherche Scientifique, Unités Mixtes de Recherche 5525,
INSERM, IFR 130, 38000 Grenoble, France (e-mail: sandrine.voros@imag.fr).
G.-P. Haber is with Cleveland Clinic, Cleveland, OH 44195 USA (e-mail:
gphaber@hotmail.com).
J.-F. Menudet is with the Endocontrol Company, 38700 Grenoble, France
(e-mail: jean-francois.menudet@endocontrol-medical.com).
J.-A. Long is with the Department of Urology, University Hospital, 38000
Grenoble, France (e-mail: JALong@chu-grenoble.fr).
P. Cinquin is with the Techniques de l’Ingénierie Médicale et de la
Complexité- Informatique, Mathématiques et Applications de Grenoble Labora-
tory, Centre National de la Recherche Scientifique, Unités Mixtes de Recherche
5525, INSERM, IFR 130, 38000 Grenoble, France, and also with the Center for
Technological Innovation (CIC-IT, INSERM), Grenoble University Hospital,
38000 Grenoble, France (e-mail: philippe.cinquin@imag.fr).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TMECH.2010.2080683
1083-4435/$26.00 © 2010 IEEE
879
manipulates the instruments, the assistant handles the laparo-
scope, thus, making the surgeon rely on his assistant for appro-
priate visualization of the surgical field and instruments. Robotic
scope holders (such as the Aesop scope holder) have been devel-
oped to overcome these limitations. They give the surgeon full
control over his visualization of the surgical field, while free-
ing the assistant’s hand. Complete telesurgery systems, such as
the daVinci robotic system (Intuitive Surgical, Sunnyvale, CA),
which offer 3-D vision and articulated instruments, are another
type of robotic system for laparoscopic surgery, which has fa-
cilitated intracorporeal surgery and increased surgeons’ inter-
est in robotics and patient acceptability of new technologies in
surgery. However, the daVinci system is bulky and expensive,
thereby, limiting its use to selected patients and procedures [2].
Thus, a current need for a robotic system less bulky and ex-
pensive was voiced. In this paper, we present the specifics of
the ViKY robotic scope holder system (Endocontrol, Grenoble,
France) and our initial clinical experience with the system. In
its current clinical use, the robot is controlled by a pedal or a
voice command. Our initial clinical experience suggested a po-
tential interest in replacing this type of “low-level” control with
a more intuitive control. Thus, an alternative control mode for
the system based on visual servoing using instrument tracking
was developed. We present here, its latest improvements and our
preliminary results of visual servoing using instrument tracking.
II. STATE OF THE ART
Robotic endoscope holders, such as the AESOP system, the
EndoAssist system [3], or more recently the body-mounted
ViKY system [4], maintain the endoscope with a robotic arm
while the surgeon performs surgery with conventional laparo-
scopic instruments. Thus, robotic endoscope holders provide a
“third hand” to the surgeon, allowing for solo surgery. They
enhance the stability and quality of the images and reduce
the staining (appearance stains on the endoscope lens due to
blood projections) of the endoscope [5]. The robotic endoscope
holders can be controlled by a vocal command or head move-
ments [6], [7], but the interactions between the surgeon and the
system remain limited (e.g., left, right, up, down, zoom in, and
zoom out).
One elegant solution aimed at developing more sophisticated
interactions between the surgeon and the system is based upon
the visual servoing of the surgical instruments. Such an ap-
proach would allow for the development of “high-level” com-
mands of the endoscope, such as the automatic displacement of
880 IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 15, NO. 6, DECEMBER 2010

the endoscope toward an area of interest pointed by a surgical

instrument, or an automatic adjustment of the zoom to main-
tain a given number of instruments in the field of view of the
endoscope. Such commands do not require a subpixel detection
of 2-D position of the tip of the instruments in the endoscopic
images, but rather a rough determination of the tip position in
order to roughly maintain an instrument in the center of the en-
doscopic view. To go one step further, it is also possible to use
a priori information about the shape of the instruments in order
to get depth information [8].
Several approaches have been proposed in the state of the
art to automatically detect instruments in endoscopic images.
Some authors have used additional equipment, such as optical Fig. 1. Body-mounted ViKY robot positioned on a patient.
localizers to localize the instruments [9] or instrument hold-
ers equipped with laser beams [10], the laser dots being more
easily detected in the endoscopic images than the actual instru-
ments. Burschka et al. [11] used kinematic information from the
daVinci system and template images of the instruments to detect
them in stereo images. Others have mounted color marks on the
instruments to facilitate their detection [8], [12], [13]. Finally,
methods purely based on image analysis have been developed:
Wang et al. [14] and Spiedel et al. [15] proposed approaches
based on color classification to isolate the pixels correspond-
ing to an instrument, and Doignon et al. [16] defined a new
color purity component that they used to robustly isolate in-
strument pixels. Finally, Climent and Marés [17] proposed an Fig. 2. ViKY control unit. (a) ViKY system console, (b) autoclavable robotic
approach based on the detection of the lines in the images, in camera holder, (c) safety foot pedal for voice control, and (d) foot pedal without
order to detect the instruments shafts, but some parameters need voice control.
to be manually tuned by the user. Moreover, apart from the
“color-marks” approaches, the proposed methods cannot deal
with several instruments in the image. We previously presented 1) The scope holder by itself: It has 3 DOF, each of them be-
a method [18] that uses the 3-D position of the instruments inser- ing actuated by autoclavable motor. This specific architec-
tion points in order 1) to identify, which instrument we want to ture allows that each of these 3 DOF exactly corresponds to
detect/track, and 2) to constrain the search for the instruments in a given direction in the image frame (left/right, up/down,
the endoscopic images, allowing for a near-real-time detection. zoom in/out). This avoids a complex geometric and cine-
The first results of the tracking of a surgical instrument with matic robot model for deriving motor commands from di-
the ViKY system on a cadaver were also reported. Following rectional information obtained in the images, which is an
initial testing of instrument tracking with the ViKY system on obvious advantage for the visual servoing task addressed
a cadaver model [18], the major steps of the protocol have been in this paper.
improved, and recent changes were brought in order to increase 2) A control unit integrating motor drivers and software for
its rapidity, robustness, and ease-of-use in a clinical context. scope holder control (see Fig. 2). One command mode
These modifications are detailed in Section IV-B. is achieved by plugging a multidirectional footswitch to
the control unit. Vocal command is an alternative choice
for scope holder control. A wireless microphone and a
voice recognition engine are used in this case. For instru-
III. CLINICALLY USED ViKY ROBOTIC ENDOSCOPE HOLDER ment tracking, video output from the camera is digitalized
ViKY is a commercially available lightweight compact endo- by a video acquisition card installed in the control unit.
scope holder. It originates from research work done at TIMC- The control unit allows the surgeon to easily redefine the
IMAG Laboratory [4]. In contrast to other endoscope holders, amplitude of the movements if required, and contains a
ViKY is entirely sterilizable (including by autoclavation), and person-specific voice-recognition training module.
can thus be attached directly to the operating table using a pas- Three versions of the ViKY exist each with different base
sive arm (see Fig. 1). The robot’s size is 115 mm (ring diameter), dimensions: regular, XL, and XS with a base of 115, 180, and
and it weights 900 g without the motors set and 2.7 kg with the 90 mm, respectively. The “regular” is compatible with most
motors set and sterilizable cables. Along with its favorable size, digestive surgeries. ViKY XS is compatible with pediatric or
sterilization capability significantly improves the global integra- thoracic surgeries and prostatectomy. ViKY XL is compatible
tion of the endoscope holder into the operating room. with single-port surgeries. Its larger base surrounds instruments
The complete system is mainly composed of two units. (see Fig. 3).
VOROS et al.: ViKY ROBOTIC SCOPE HOLDER: INITIAL CLINICAL EXPERIENCE AND PRELIMINARY RESULTS
Fig. 3. ViKY XL during a single-port surgery.
The ViKY System received EC marking in 2007 and FDA
approval in 2008. Since then, over 1000 surgical procedures
have been performed with ViKY. Section V-A will detail the
results obtained during this two-year period.
In its current clinical use, the ViKY system is controlled by
the surgeon using a pedal or a vocal command, allowing for:
1) basic displacements (left, right, up, down, zoom in, and
zoom out) with a large amplitude;
2) basic displacements with a small amplitude; and
3) memorization of a robot position of interest and automatic
return to this position.
When disabled, ViKY can be freely moved by hand in any
direction (right/left, up/down, and zoom).
In an attempt to propose a more intuitive interaction between
the surgeon and the ViKY system, we previously described
[18] a method for displacing the endoscope by maintaining a
selected instrument roughly centered in the field of view of the
endoscope (tool tracking). We present in the next section, its
general framework and the improvements carried out following
a cadaver experiment.
IV. AUTOMATIC DETECTION OF THE SURGICAL INSTRUMENTS
USING THEIR INSERTION POINTS
A. General Framework
The overall framework of our tool-tracking method is sum-
marized in Fig. 4. The improvements carried out at each step
are then detailed in Section IV-B.
We assume that both the ViKY robot and the imaging system
are calibrated: the 2-D coordinates of a point in the endoscopic
image can be computed using its 3-D coordinates in a fixed
“world frame” linked to the robot. The imaging system (en-
doscope) was calibrated using the commonly used the Zhang
method [19] with a calibration grid. The practical calibration
protocol, compatible with clinical constraints, is presented in
Section IV-B-1).
In the first step, the surgeon measures the positions of the
instruments’ insertion points (yellow box in Fig. 4). Previously,
this was done with the calibrated ViKY system: given a cal-
ibrated ViKY system, we asked the surgeon to indicate the
position of the insertion point of each instruments in the endo-
scopic images for two positions of the ViKY robot. The 3-D
881
Fig. 4. Framework for the detection and tracking of surgical instruments
using a priori information about the position of their insertion points and image
analysis. The yellow boxes correspond to the initialization step, the grey boxes
correspond to the actual segmentation of the surgical instruments, and the blue
boxes correspond to the visual servoing of the ViKY robot.
(fixed) position of the insertion point could then be computed
using stereovision. This approach was a bit cumbersome and a
simplified approach, easier to use in clinical conditions, is pre-
sented in Section IV-B2. Doignon et al. [20] also developed a
method allowing the automatic computation of the position of
the insertion points by performing “pivot” movements with the
instruments.
In the second step (grey box in Fig. 4), the 3-D position of
the insertion point corresponding to the instrument we wish to
track is projected on the image plane to constrain the search for
the instrument in the image. This constraint is used to detect
the lines corresponding to the edges of the instrument in the
image, and then the symmetry axis of the instrument in the
image. Compared to our previous method [18], we improved the
detection of the symmetry axis by integrating it in a probabilistic
framework. This improvement is presented in Section IV-B-3.
Once the symmetry axis of the instrument is found, we search for
the tip of the instrument along the axis using color information
(see Fig. 5).
Finally, the 2-D position of the instrument in the image is
inputted into a visual servoing loop in order to automatically
move the endoscope toward the tip of the instrument (blue box
in Fig. 4).
B. Improvements of the Method
1) Calibration Protocol: The calibration step aims at es-
timating the intrinsic and extrinsic parameters of the system
{ViKY robot + endoscope}: given a 3-D point P expressed in a
fixed world referential linked to the robot (frame R0 in Fig. 6),
we need to be able to compute P’s coordinates in a mobile ref-
erential linked to the imaging system (frame RC in Fig. 6), i.e.,
we need to determine the transformation matrix T between R0
and RC , which corresponds to the extrinsic calibration. We also
need to know the characteristics of our imaging system, i.e.,
the transformation C that projects the point P, expressed in the
882
Fig. 5. Detection of two surgical instruments from a recording of real surgeries
using the proposed method. The detection process does not rely on the color of
the surgical instrument.
Fig. 6. Presentation of the three referentials involved in the calibration of the
{ViKY + imaging system}, and the two calibration matrices that need to be
determined (C: intrinsic calibration, T: extrinsic calibration).
mobile frame, into a 2-D point p in the image. This corresponds
to the intrinsic calibration. This calibration step is mandatory
in order to exploit, in the endoscopic images, the constraints
imposed by the instruments’ insertion points.
For the determination of the intrinsic parameters, a classical
plane-based calibration approach is used. The surgeon moves
the camera while looking at a calibration plane, such as the one
shown in Fig. 7. The grid corners are automatically extracted
using a standard corner detection algorithm and used to com-
pute the intrinsic parameters of the imaging system [19] with
OpenCV computer vision library [21]. The user is informed
when enough data has been extracted from the images for in-
trinsic calibration (usually, a 20-s sequence provides enough
data, which corresponds to ten images of the grid).
For the extrinsic parameters, we assume 1) a fixed world
referential R0 2), a mobile referential attached to the robot,
and 3) the camera coordinate frame RC . At robot initialization,
world and robot frames are assumed to be identical. Motor
encoders and geometrical model of the robot allow estimating
the rigid transformation, relating these two coordinate frames for
any subsequent position of motors. Therefore, the only unknown
is the relationship between the robot and camera frames.
To simplify the problem, we can choose the robot coordinate
frame such that the transformation relating it to the camera frame
IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 15, NO. 6, DECEMBER 2010
Fig. 7. Planar calibration grid (square size = 8 mm) used to calibrate the
{ViKY + imaging system}.
Fig. 8. Calibration object used for ViKY’s extrinsic calibration.
is a simple combination of a translation along the optical axis
and a rotation around the same axis. Thus, only two parameters,
translation magnitude and rotation angle, have to be estimated.
In practice, they are estimated using a calibration object shown
in Fig. 8, reproducing endoscope positioning in ViKY. The en-
doscope looks at a planar calibration grid of known size, and,
the camera being calibrated, a single image of this grid gives the
two unknown parameters from basic geometrical computations.
2) Determination of the Positions of the Insertion Points:
The protocol for determining positions of the insertions points
is again relatively easy. The first step is to move the zooming
rod of ViKY in the direction of the insertion point of interest
(with the camera and endoscope installed on ViKY). Using the
motor encoders and geometrical model of the robot, we thus
approximately know the orientation (θ, ϕ) of the insertion point
in spherical coordinates. The last coordinate ρ is measured on
the patient assuming that the insertion point lies in the plane of
the ViKY base, which is a reasonable approximation when the
insertion point is not too far away from the robot. In practice, this
approximation leads to a very simple protocol. We will see in
the discussion that a more precise determination of the insertion
points could be performed, while still taking into account the
clinical constraints.
3) Probabilistic Framework: In our previous work [18], we
used the a priori information about the insertion points in order
to constrain the search for lines in the image that could corre-
spond to the edges of the instrument we wished to detect/track.
This detection was performed frame by frame, without exploit-
ing the knowledge acquired about the tool’s pose in the previous
frames. Moreover, in difficult conditions (for instance, low con-
trast because of smoke, or a lot of noise because of specular
reflections on the tools), the detection could be time consum-
ing without a maximum detection time, hence, degrading the
tracking. We now have integrated our tracking method into a
VOROS et al.: ViKY ROBOTIC SCOPE HOLDER: INITIAL CLINICAL EXPERIENCE AND PRELIMINARY RESULTS 883

Fig. 9. 2-D parameterization of an instrument’s axis constrained by the posi-

tion of the projection I of its insertion point in the endoscopic image plane. θ
represents the orientation of the line and ρ its distance to I.

Fig. 11. Framework of the Condensation algorithm used to detect the instru-
ment’s axis in the endoscopic images.

We then use the iterative condensation algorithm [22] in order

Fig. 10. Variation space of the two parameters θ and ρ. to determine the position of the instrument (see Fig. 11). The
scores of the measurement step are obtained by measuring, for
each particle in the set, the number of nonnull gradients that the
probabilistic framework, in order to take into account these lim-
line the particle represents intersects.
itations.
Once the axis of the instrument is determined, the search
As illustrated by Fig. 9, a surgical instrument in the endo-
for the tip is performed using an Otsu threshold [23] on the
scopic image can be represented by a 2-D line in the neighbor-
points of the instrument’s axis in order to find the threshold that
hood of the projection I of the instrument’s insertion point on
maximizes the interclass variance. The pixels belonging to the
the image plane. The line can be parameterized in the Hough
axis are then studied, starting with the point in the image, which
space centered on I by two parameters θ and ρ, respectively,
is the closest to the insertion point, in order to determine the
representing the orientation of the line and its distance to the
position of the tip.
insertion point. The variation space of θ is constrained in [θm in ,
θm ax ] by the position of the insertion point with respect to the
endoscopic image, and the variation space of ρ is constrained V. RESULTS
in [ρm in , ρm ax ] by the apparent diameter of the tool in the im- The results presented in this section are three fold: we first
age and an adjustable threshold that can take into account the present our clinical experience with the version of the ViKY
imprecisions in the determination of the insertion point (see system authorized for clinical use (with pedal or voice com-
Fig. 10). mand). While the robot clinical use was studied, a new surgical
Given this parameterization, we can now integrate the search technique called “single-port” surgery appeared, as well as a
for the instrument’s axis into a probabilistic framework. need on our side to ensure that the ViKY system would be com-
We represent a line by the following 4-D particle: patible with this trend. Thus, we secondly present the results
⎡ ⎤ of our animal pilot study and first clinical cases in the single-
ρ(t − 1) port configuration. Finally, we present our preliminary results in
⎢ θ(t − 1) ⎥
⎣ ⎦. our ongoing testing of the tool-tracking command of the ViKY
ρ(t) system.
θ(t)
The incorporation of the particle’s state at the previous frame A. Clinical Experience With the ViKY System
allows us to predict the position of the particle at the next frame. 1) Traditional Laparoscopic Surgery: The first surgery us-
Assuming a linear movement, we can model the displacement ing the VIKY robot on a patient was performed on July 5, 2007,
of a particle as follows: in the Urology Department of Grenoble University Hospital. The
⎡ ⎤ ⎡ ⎤ ⎡ ⎤ procedure selected for this first patient validation was a pelvic
ρ(t) 0 0 1 0 ρ(t − 1)
⎢ θ(t) ⎥ ⎢ 0 0 0 1 ⎥ ⎢ θ(t − 1) ⎥ laparocopic lymphadenectomy for prostate cancer staging. The
⎣ ⎦=⎣ ⎦ . ⎣ ⎦ robot was controlled using vocal recognition. Operative time
ρ(t + 1) −α 0 1 + α 0 ρ(t)
θ(t + 1) 0 −β 0 1+β θ(t) was 145 min including the robot positioning, compared to ap-
proximately 120 min without the robot. No intraoperative com-
where α and β are dumping coefficients in [0, [1] that are plications were noted. No unsolicited movement of the robot
empirically determined. was recorded. In this intervention, the length of the surgery with
884
TABLE I
DESCRIPTION OF THE ROBOTIZED PROCEDURES USING THE VIKY SYSTEM
the robot was slightly superior to the one without the robot. This
can be explained by the fact that the surgeons were discovering
the system and learning to use it.
After this success, a multicenter prospective study has been
launched so as to prove the efficiency of the robot system for
laparoscopic procedures such as radical prostatectomy, radical
nephrectomy, pyeloplasty, sacral colpopexy, and cholecystec-
tomy. The study is still in process in three French centers.
Preliminary results on 53 patients (30 prolapse surgeries, 3
nephrectomies, 6 cholecystectomies, 2 pyeloplasties, 12 prosta-
tectomies) show that most of the procedures can be performed
with the robot. The installation time is approximately 5 min. No
perioperative complications related to the robot were recorded.
The mean operative time for each procedure is described in
Table I. The prospective study being still in process, we cannot
yet compare the operative time to classical laparoscopic surgery,
but the surgeons who performed the surgeries with the robot did
not note a drastic difference in operating time.
The major limitation observed was the lack of amplitude
of motions that appeared during colonic dissection for kidney
surgery, which may in some cases require a momentary help
of the assistant to hold the camera, or a switch to classical
laparoscopy (see Table I). The major advantage of the robot
stands in the freeing of the assistant hand by the robot.
2) Single-Port Laparoscopic Surgery: The ViKY robot was
also tested in a single-port configuration on five male farm pigs.
Bilateral partial nephrectomy and bilateral pyeloplasty was per-
formed through a single port before the completion of bilateral
radical nephrectomy. There were no intraoperative complica-
tions, and there was no need for additional ports to be placed.
The mean (range) operative duration for partial nephrectomy,
pyeloplasty, and nephrectomy was 120 (100–150), 110 (95–
130), and 20 (15–30) min, respectively. The mean (range) esti-
mated blood loss for all procedures was 240 (200–280) mL. The
preparation time decreased from 25 to 15 min after 10 cases (p
= 0.002) [24].
Following the animal pilot study, the base of the robot was
enlarged and adapted to the specifics of single port, and this
version of the VIKy was called XL. The VIKY XL was then used
for single-port procedures in five patients undergoing radical
nephrectomy. A 5-mm, 30◦ scope was used, robot positioning
did not differ from the Standard ViKY, however, installation was
found to be 10 min longer to be able to optimize the range of
movement of the instruments avoiding clutching with the robot.
IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 15, NO. 6, DECEMBER 2010
Limitation of the movement was found during the dissection
at the upper pole of the kidney, and while dropping the colon
at the level of the pelvic brim. The amplitude of movements
was limited, and external conflict between the scope and the
instruments was found, and required the intervention of the
assistant to compensate the lack of rotation that is needed when
using a 30◦ scope in single-port surgery.
The next step for the system is the instruments tracking to
guide the endoscope. Our preliminary experience with this guid-
ance modality during cadaver and pig experiments let us think
that it would improve the comfort and the efficiency of the
procedures.
B. Toward an Image-Analysis-Based Command of the ViKY
System
The quantitative evaluation of the tool-tracking method is
still a work in progress. We present here two aspects regarding
the accuracy of the system that were quantitatively evaluated:
calibration accuracy, and the validation of the “fixed” insertion
points hypothesis. As of today, we have not quantitatively eval-
uated the precision and robustness of the method, although, we
can qualitatively say that it was dramatically improved com-
pared to our original implementation of the method.
1) Quantitative Evaluation of the Calibration Accuracy: We
evaluated the accuracy of the intrinsic calibration of the system
by using the rms reprojection error metric classically used with
Zhang’s grid-calibration method. We used ten images of the
calibration grid to compute the intrinsic calibration parameters.
These parameters were then used to reproject the 3-D coordi-
nates of the grid corners in the images of the grid, and compared
to the 2-D grid corners automatically extracted using image
analysis. The global reprojection error for the image set was
1.06 pixels (on images of a 720 × 480 resolution).
We then evaluated the accuracy of the overall calibration of
the system (intrinsic + extrinsic). To do so, we took an image of
the calibration grid for a position p1 of the ViKY robot. Then,
we moved the robot to another position p2 with the calibration
grid still visible. Using motor encoders, geometrical model of
the robot, and precomputed transformation between robot and
camera frames, the geometric transformation T between the
camera frames c1 and c2 corresponding to positions p1 and p2
was computed (i.e., calibration grid images were not used in this
computation). Using T, we expressed the 3-D coordinates of the
grid corners in c2 and projected them on the image plane. These
2-D coordinates were then compared to the 2-D grid corners
automatically extracted using image analysis. The reprojection
error for one grid image of size 720 × 480 pixels was 49.4
pixels. This result shows that the extrinsic calibration is approx-
imative (probably due to a poor robot geometrical model), but
we use the calibration only to roughly determine the position of
the insertion points, which then gives instrument orientation. In-
sertion points being generally outside and far away from image,
instrument orientation is actually not strongly affected by such
an error. We will see in the next section that this approxima-
tive calibration is sufficient for the determination of the tool’s
position within our precision requirements.
VOROS et al.: ViKY ROBOTIC SCOPE HOLDER: INITIAL CLINICAL EXPERIENCE AND PRELIMINARY RESULTS
TABLE II
MEASUREMENTS ALONG THE TIME OF THE POSITIONS OF AN INSTRUMENT’S
INSERTION POINT DURING A CADAVER EXPERIMENT
2) Quantitative Validation of the Hypothesis That the Instru-
ments Insertion Points can be Considered as “Fixed” in the
Robot Referential: In our method, we use the insertion points
to constrain the search for the instruments in the image. We do
not require a very accurate knowledge of the insertion points, but
we assume that they are relatively “fixed” during the whole sur-
gical intervention. To validate this hypothesis, we performed a
cadaver test, in which we measured, using stereovision, the 3-D
positions of an instrument’s insertion point in the fixed robot ref-
erential at several times. Seven measurements were performed
in 235 min (see Table II). On an average, the error was less than
5 mm, except for one measurement, for which there was a leak
of gas in the abdominal cavity.
We have not yet evaluated in a standalone test, the precision of
the measurement of the insertion points. For now, we have only
evaluated the final precision of the method in the images, i.e.,
compared the tool’s tip position found with our tool-detection
method to the one manually clicked a posteriori on the images
recorded during a cadaver experiment.
3) Qualitative Evaluation of the Precision of the Method:
The method was first tested on several videos from real proce-
dures. This step allowed us to prevalidate our tool segmentation
approach (with several instruments in the image, with instru-
ments of different colors, etc.), and identify the difficult cases.
Using our initial implementation of the method, the detec-
tion time was on an average 100 ms for a subsampled image of
200 × 100 pixels. Since this first experiment, detection time has
been dramatically improved using the probabilistic framework
described previously. For comparison, average detection time
for an image of 480 × 360 pixels is now 300 and 40 ms for
200 × 100 pixels. In 70% of the cases, the image error was
inferior to 5 pixels, and in 87% of the cases, inferior to 11 pix-
els. According to the surgeons who performed the experiments,
this precision is sufficient for our purpose of orienting the en-
doscope. Indeed, when a surgeon performs a very “fine” tool
displacement, often corresponding to precision work, the cam-
era must be motionless. It is only for larger tool displacements
that the surgeon might want the endoscope to follow the tool’s
displacement.
Our next step is to quantitatively evaluate the precision and
robustness of our new implementation on several videos taken
in conditions close to the clinical reality. We can already state
that the robustness of the method seemed to be improved, since
we were able to track an instrument in sequences, where the
initial method failed (strong specular reflections).
4) Qualitative Evaluation During a Pilot Animal Experi-
ment: A pilot study on animal model allowed us to test our
885
tool-tracking improvements in conditions close to the clinical
reality. During these experiments, we empirically chose a rele-
vant number of particles (2000) and fine-tuned the parameters
θm in , θm ax , ρm in , ρm ax , α, and β. We were able to successfully
track the tip of a selected instrument (when only this instrument
was present in the endoscopic image), and observed that the
tracking was lost when the surgical instrument passed near an
organ that had long and straight edges (like the liver, or line of
Toldt). However, when the surgical instrument was lost, its tip
was rapidly recovered by moving the instrument slowly in front
of the endoscope.
Finally, we were able to test our calibration protocol at the
beginning of five procedures (laparoscopic kidney and adrenal
surgery). The robot was installed with an average time of 5 min.
A standard port-placement configuration was used and a 0◦ ,
12-mm scope was positioned between the two instruments port.
The distance between the camera port and the right port with the
dominant instrument was recorded. Time needed for a successful
calibration was 10 min. The images of the interventions, as well
as the robot’s coordinates, were saved for a future retrospective
analysis of the instruments’ detection.
VI. DISCUSSION/FUTURE WORK
In this paper, we presented the clinical experience with a
lightweight, sterilizable robotic endoscope holder. The use of
the robot did not significantly increase the duration of the sur-
gical procedures, and its installation time decreased with ex-
perience (no more than 5 min for experienced users). The first
clinical experiments with the robot for single-port technique
were also encouraging. We are now waiting for the results of
our multicenter prospective clinical study of the system to assess
quantitatively its potential clinical added value.
One limitation concerning the endoscope’s amplitude of
movements was found during kidney surgery. A new design
of the endoscope holder is currently under development to over-
come this limitation. It will allow wider movement range and
will also free space around the endoscope, which is a crucial
issue for single-port procedure. This could also be resolved in
the future using a 30◦ angle scope and the addition of a 4 DOF
allowing a rotation of the scope.
While this clinical study was taking place, we improved our
tool-tracking method, by adapting the calibration procedure to
the clinical constraints, and by integrating it to a probabilis-
tic framework. We were able to validate the feasibility of the
calibration in clinical conditions. The measurement of the inser-
tion points was performed approximately, since the parameters
of our method allow us to take into account the measurement
errors. However, the more imprecise the measurement of the
insertion points is, the bigger the search space is, with obvious
consequences on the computation time. In the future, we plan:
1) on improving the precision of the calibration step (for in-
stance by performing a hand-eye calibration [25]); and 2) on
measuring, at the beginning of the intervention, the positions of
the insertion points by performing “pivot” movements with the
instrument inside the abdominal cavity, as described by Doignon
et al in [20]. Also, the insertion point’s position could be updated
886
intraoperatively using the results of the instrument’s detection.
This would be important if the positions of the insertion points
change by modifying the pneumoperitoneum pressure. With
the patient’s abdomen being insufflated with gas, the effects
of respiration should be minimal. Further, we did not observe
significant displacements of the insertion points during our pig
experiments.
The integration into a probabilistic framework allowed us
to take into account the position of the tip of the instruments
in previous frames instead of working frame-by-frame, thus
increasing the speed of the detection as shown by the results
presented in Section V-B. The feasibility of this new approach
was validated during animal experiments, in which, we observed
two main reasons for a tracking failure.
1) Specular reflections on the organs. We are currently inves-
tigating two solutions for specular reflections. A software-
based removal (but it is very costly in computation time);
solutions based on polarizing filters, but they decrease the
illumination of the scene [26].
2) Presence of organs with long and straight edges. We are
currently investigating a solution that would consist in
measuring patient-specific color information at the begin-
ning of the procedure, in order to remove edges corre-
sponding to organs in the search for the instrument.
VII. CONCLUSION
Our clinical experience with a lightweight robotic endoscope
holder is very encouraging. Indeed, such a system can provide
a “third hand” to the surgeon during laparoscopy and a viable
option for single-port surgery. Preliminary results of instrument
tracking based on image analysis demonstrated the feasibility
and the safety of this approach in preclinical conditions. Lim-
itations were identified and solutions are being investigated. A
prospective randomized study comparing the use of the ViKY
to conventional practice is ongoing. This technology, once fine
tuned, has the potential to take robotic surgery to a new level
by letting the robot intelligently assist the surgeon, while allow-
ing the surgeon to focus on the surgical act rather than on the
interaction with the robot.
REFERENCES
[1] B. Makhoul, A. De La Taille, D. Vordos, L. Salomon, P. Sebe, J. Audet,
L. Ruiz, A. Hoznek, P. Antiphon, A. Cicco, R. Yiou, D. Chopin, and C.
Abbou, “Laparoscopic radical nephrectomy for T1 renal cancer: The gold
standard? A comparison of laparoscopic vs open nephrectomy,” BJU Int.,
vol. 93, no. 1, pp. 67–70, 2004.
[2] J. C. Hu, X. Gu, S. R. Lipsitz, M. J. Barry, A. V. D’Amico, A. C. Weinberg,
and N. L. Keating, “Comparative affectiveness of minimally invasive vs
open radical prostatectomy,” J. Amer. Med. Assoc. (JAMA), vol. 302,
no. 14, pp. 1557–1564, 2009.
[3] P. Ballester, Y. Jain, K. R. Haylett, and R. F. McCloy, “Comparison of
task performance of robotic camera holders EndoAssist and Aesop,” Int.
Congr. Ser., vol. 1230, pp. 1100–1103, Jun. 2001.
[4] J.-A. Long, P. Cinquin, J. Troccaz, S. Voros, P. Berkelman, J.-L. Descotes,
C. Letoublon, and J.-J. Rambeaud, “Development of miniaturized light
endoscope holder robot for laparoscopic surgery,” J. Endourol., vol. 21,
no. 8, pp. 911–914, Aug. 2007.
IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 15, NO. 6, DECEMBER 2010
[5] L. R. Kavoussi, R. G. Moore, J. B. Adams, and A. W. Partin, “Comparison
of robotic versus human laparoscopic camera control,” J. Urol., vol. 154,
no. 6, pp. 2134–2136, Dec. 1995.
[6] P. A. Finlay, “Clinical experience with a goniometric head-controlled
laparoscope manipulator,” in Proc. IARP Workshop Med. Robot., Vienna,
Austria, Oct. 1996.
[7] A. Nishikawa, T. Hosoi, K. Koara, D. Negoro, A. Hikita, S. Asano, F.
Miyazaki, M. Sekimoto, Y. Miyake, M. Yasui, and M. Monden, “Real-
time visual tracking of the surgeon’s face for laparoscopic surgery,” Med.
Image Comput. Comput.-Assisted Intervention Conf. (MICCAI), 2001,
Lecture Notes Comput. Sci., vol. 2208, pp. 9–16, 2001.
[8] O. Tonet, T. U. Ramesh, G. Megali, and P. Dario, “Image analysis-based
approach for localization of endoscopic tools,” in Proc. Surgetica, 2005,
pp. 221–228.
[9] P. Berkelman, E. Boidard, and P. Cinquin, “Automatic instrument tracking
with a compact laparoscopic endoscope robot using an external optical
localizer,” in Proc. Surgetica, 2002, pp. 17–24.
[10] A. Krupa, J. Gangloff, C. Doignon, M. de Mathelin, G. Morel, J. Leroy,
L. Soler, and J. Marescaux, “Autonomous 3-D positioning of surgical
instruments in robotic laparoscopic surgery using visual servoing,” IEEE
Trans. Robot. Autom., vol. 19, no. 5, pp. 842–853, 2003.
[11] D. Burschka, J. J. Corso, M. Dewan, W. Lau, M. Li, H. Lin, P. Marayong,
N. Ramey, G. D. Hager, B. Hoffman, D. Larkin, and C. Hasser, “Navigating
inner space: 3-D assistance for minimally invasive surgery,” Robot. Auton.
Syst., vol. 52, no. 1, pp. 5–26, 2005.
[12] G. Wei, K. Arbter, and G. Hirzinger, “Real-time visual servoing for laparo-
scopic surgery: Controlling robot motion with color image segmentation,”
IEEE Eng. Med. Biol. Mag., vol. 16, no. 1, pp. 40–45, Jan./Feb. 1997.
[13] A. Casals, J. Amat, and E. Laporte, “Automatic guidance of an assistant
robot in laparoscopic surgery,” in Proc. IEEE Int. Conf. Robot. Autom.,
Apr. 1996, vol. 1, pp. 895–900.
[14] Y. Wang, D. R. Uecker, and Y. Wang, “A new framework for vision
enabled and robotically assisted minimally invasive surgery,” Comput.
Med. Imaging Graph., vol. 22, no. 6, pp. 429–437, 1998.
[15] S. Spiedel, G. Sudra, S. Schalck, B. P. Müller-Stich, C. Gutt, and R.
Dillman, “Automatic image-based analysis for context-aware assistance
in minimally invasive surgery,” in Proc. Surgetica, 2007, pp. 53–62.
[16] C. Doignon, P. Graebling, and M. de Mathelin, “Real-time segmentation
of surgical instruments inside the abdominal cavity using a joint hue
saturation color feature,” Real-Time Imaging, vol. 11, no. 5–6, pp. 429–
442, 2005.
[17] J. Climent and P. Marés, “Automatic instrument localization in laparo-
scopic surgery,” Electron. Lett. Comput. Vis. Image Anal., vol. 4, no. 1,
pp. 21–23, 2004.
[18] S. Voros, E. Orvain, J.-A. Long, and P. Cinquin, “Automatic detection
of instruments in laparoscopic images: A first step towards high level
command of robotized endoscopic holders,” Int. J. Robot. Res., vol. 26,
no. 11–12, pp. 1173–119, 2007.
[19] Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans.
Pattern Anal. Mach. Intell. (PAMI), vol. 22, no. 11, pp. 1330–1334, 2000.
[20] C. Doignon, F. Nageotte, and M. de Mathelin, “Segmentation and Guid-
ance of Multiple Rigid Objects for Intra-Operative Endoscopic Vision,”
in Proc. Workshop on Dynamical Vision of ECCV, , 2005/2006, Lecture
Notes Comput. Sci., vol. 4358, pp. 314–327, 2007.
[21] [Online]. Available: http://sourceforge.net/projects/opencvlibrary/
[22] M. Isard and A. Blake, “Condensation—Conditional density propagation
for visual tracking,” Int. J. Comput. Vis., vol. 29, no. 1, pp. 5–28, 1998.
[23] N. Otsu, “A threshold selection method from gray level histograms,” IEEE
Trans. Syst., Man Cybern., vol. 9, no. 1, pp. 62–66, Jan. 1979.
[24] S. Crouzet, G. P. Haber, W. M. White, K. Kamoi, R. K. Goel, and
J. H. Kaouk, “Single-port, single-operator-light endoscopic robot-assisted
laparoscopic urology: Pilot study in a pig model,” BJU Int., vol. 105,
no. 5, pp. 682–685, Mar. 2010.
[25] F. Dornaika and R. Horaud, “Simultaneous robot-world and hand-eye
calibration,” IEEE Trans. Robot. Autom., vol. 14, no. 4, pp. 617–622,
Aug. 1998.
[26] J. Duchateau, S. Voros, and P. Cinquin, “Development and precision eval-
uation of an image-analysis based endoscopic tool tracking system,” pre-
sented at the Surgetica Conf., Chambéry, France, Sep. 19, 2007.
Authors’ photographs and biographies not available at the time of publication.