You are on page 1of 11

Minimally Invasive Therapy.

2010;19:211

REVIEW ARTICLE

How to objectively classify residents based on their psychomotor


laparoscopic skills?

MAGDALENA K. CHMARRA1, CORNELIS A. GRIMBERGEN1,2,


FRANK-WILLEM JANSEN1,3, JENNY DANKELMAN1
1
Department of BioMechanical Engineering, Delft University of Technology, Delft, the Netherlands, 2Department of
Biomedical Engineering & Physics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands,
and 3Department of Gynecology, Leiden University Medical Center, Leiden, the Netherlands

Abstract
In minimally invasive surgery (MIS), a surgeon needs to acquire a certain level of basic psychomotor MIS skills to perform
surgery safely. Evaluation of those skills is a major impediment. Although various assessment methods have been introduced,
none of them came as a superior. Three aspects of assessing psychomotor MIS skills are discussed here: (i) advantages and
disadvantages of currently available assessment methods, (ii) methods to objectively classify residents according to their level of
psychomotor skills, and (iii) factors that inuence psychomotor MIS skills. Motion analysis has a potential to be the means to
deal with assessment of psychomotor skills. Together with classication methods (e.g. linear discriminant analysis), motion
analysis provides an aid in deciding whether a resident is ready to move to the next level of training. Presence of factors that
inuence psychomotor MIS skills results in a high need for standardisation of valid tasks and setups used for the assessment of
MIS skills.

Key words: Laparoscopy, psychomotor skills, training, assessment, motion analysis

Introduction objective assessment of technical performance is in its


infancy (11).
It is indisputable that a surgeon needs to acquire a Various scoring methods to assess laparoscopic skills
certain level of manual dexterity to perform surgery have been developed (12-14). However, widely used
safely. Therefore, surgical organisations (e.g., methods to objectively determine whether a resident
Accreditation Council of Graduate Medical can be called an expert, intermediate, or novice
Education ACGME) are calling for methods to (according to his/her laparoscopic skills) are still miss-
ensure the maintenance of skills, advance surgical ing. It is, therefore, necessary to develop such objective
training, and to credential surgeons as technically assessment methods and to implement them in the
competent (1-3). A variety of training simulators is training curriculum. Ideally, objective assessment
commercially available and used to train basic psy- methods address at least two topics: They identify
chomotor skills needed in minimally invasive surgery potential areas of improvement, and they determine
(MIS, e.g., laparoscopy) (4-10). Since the introduc- whether a resident is competent to move to the next level
tion and implementation of those simulators in the of training (e.g. to the operating room (OR)).
curriculum of surgeons training, there is growing Surgery, and MIS in particular, requires a broad
consensus about the need to assess surgical compe- range of skills (e.g. technical skills, motor skills, cog-
tence objectively. Evaluation of MIS skills, however, is nitive knowledge, surgical judgement, fast acting,
still a major impediment. Although progress has been team work, communication skills) (15, 16). Hence,
made over the past years, at this stage, the science of various objective assessment methods are needed to

Correspondence: M. K. Chmarra, Department of BioMechanical Engineering. Faculty of Mechanical, Maritime and Materials Engineering (3mE),
Delft University of Technology, Mekelweg 2, 2628 CD Delft, the Netherlands, Fax: 31-15-27 84 717; E-mail: m.k.chmarra@tudelft.nl

ISSN 1364-5706 print/ISSN 1365-2931 online 2010 Informa UK Ltd


DOI: 10.3109/13645700903492977
Objective classication 3

assess those skills. This paper will elaborate only on Skills OSATS (13, 24-28)) have gained a lot of
topics related to the assessment of basic psychomotor attention (29, 30). These methods have been vali-
MIS skills. Therefore, advantages and shortcomings dated in training setups (e.g. skills labs) and showed
of current assessment methods will be given. Further, that they can be of great value (24-28). However, their
methods to classify residents as experts, inter- validation and implementation in the OR is difcult,
mediates, and novices, according to their basic since it is not clear when the various scores should be
psychomotor MIS skills, will be discussed. Examples given. It is, e.g., difcult to judge the skills of the rst-
of factors that inuence psychomotor MIS skills will year resident and the fth-year resident, who both
be given. Finally, a number of recommendations and obtained the highest OSATS score, because during
possible approaches to improve current assessment of assessment, the evaluator often takes into account
residents basic psychomotor skills will be proposed. their year of the residency. As a consequence, both
residents have the same score, but their MIS skills are
different.
Assessment of basic psychomotor MIS skills In many hospitals and training centres, assessment
of MIS skills is often done by surgical educators, who
Presently, one of the most essential and widely can be inuenced by, e.g., personal relationships.
accepted objective measure of residents competence Since this kind of assessment is per denition not
is the number of performed cases (17). This measure objective (30, 31), the emphasis is put on using
is easily quantiable (Table I). However, it does not trainers (e.g., box trainers and virtual reality (VR)
represent the real competence of a resident. For trainers) for the assessment of individual skills (e.g.,
example, it has not yet been investigated how many knowledge of steps of a procedure, basic psychomotor
performance cases are required for competency. MIS skills) (32-34).
Moreover, it is very likely that different individuals Various studies showed that basic psychomotor
require different numbers of cases for gaining com- MIS skills can successfully be assessed by analyzing
petency (18). motions of MIS instruments (12, 30). Figure 1 shows
Besides the number of performed procedures, the an example of typical MIS instrument trajectories of
most frequently used objective evaluation measure of an expert and a novice resident performing the same
residents MIS skills in the OR is the number of task. A range of measures have been proposed. All of
complications (19-23). This measure is also easily them are parameters based on time-dependent three-
quantiable (Table I). However, such assessment of dimensional (3D) representation of the tip motions of
MIS skills based on morbidity and mortality data fails the MIS instrument together with the rotation of the
to represent the actual residents competence, since instrument around its axis:
each patient and case are always different and cannot
be easily compared. . time total time taken to complete the task
Evaluation methods based on task-specic check- . path length the length of the curve described by
lists and global ratings scales (e.g., Global Operative the tip of the instrument
Assessment of Laparoscopic Skills GOALS, and . movement economy ideal path length divided by
Objective Structured Assessment of Technical the actual path length, where the ideal path length is
the straight-line distance between two targets
. deviation from the ideal path the sum of the
Table I. Overview of assessment methods
differences between the actual and the ideal path
Assessment of . depth perception total distance travelled by the
Training psychomotor instrument along its axis
Objectivity environment MIS skills . accuracy the correctness of placing the tip of the
No. Cases + OR - instrument in 3D
. rotational orientation the amount of rotation of
No. Complications + OR -
the instrument around its axis
GOALS - OR, box, VR +/- . angular area parameter related to the distances
OSATS - OR, box, VR +/- between the farthest left-right and forward-back-
Educator - OR, box +/- ward positions of the instrument while performing
Motion analysis + VR, box + a task
. volume parameter related to the distances
MIS minimally invasive surgery; OR operation room; box box
between the farthest left-right, forward-backward,
trainer; VR virtual reality; GOALS Global Operative Assess-
ment of Laparoscopic Skills; OSATS Objective Structured and in-out positions of the instrument while per-
Assessment of Technical Skills. forming a task
4 M. K. Chmarra et al.

Expert Novice

-140
-140

zt [mm]
-160
zt [mm]

-160
-180
-180
-200
-200
40 0
40 20
20 0 0 -50
0 yt [mm] -20
-50 -40 -100 xt [mm]
yt [mm] -20
-40 xt [mm]
-100

Figure 1. An example of typical MIS instrument trajectories of an expert and a novice resident performing a positioning task with the left hand.
The camera was held by the resident in his/her right hand.

. motion smoothness parameter which represents a Consequently, there is no agreement whether a


change of the acceleration, etc. (12, 35-41). resident should be assessed on individual tasks, or
whether a composite assessment of all tasks should be
used (46).
Classication based on basic psychomotor MIS There is a shortage of methods that can be used to
skills distinguish between residents of varying experience,
especially when ne gradations in experience need to
There is a tendency to assess residents performance
be detected. The American Board of Surgery and the
in MIS using a few very basic aspects. Mostly, those
Royal Australasian College of Surgeons have both
are: Time, path length, and penalty points. Such
mandated Fundamentals of Laparoscopic Surgery
assessment, however, does not show the actual size
(FLS) certication of candidates before they receive
and the nature of the gap between expert and novice
their specialty certication. The FLS is a program
performance (12). From the clinical point of view, it
which was developed by the members of the Society of
is important to objectively determine (classify)
American Gastrointestinal and Endoscopic Surgeons
whether a resident can be called an expert, intermediate,
(SAGES) and the American College of Surgeons
or novice (according to his/her MIS skills). In daily
(ACS) in 2004 (47, 48). The technical skills of resi-
practice, it has been recognised that one performance
dents are being evaluated in the FLS based on the
measure alone (e.g. time) does not sufciently mea-
McGill Inanimate System for Training and Evalua-
sure basic MIS skills. Therefore, basic MIS skills are
tion of Laparoscopic Skills (MISTELS) score
often assessed using at least two different outcome
(47, 49). The score is computed for each individual
measures (42-44). It has also been found that perfor-
task and takes only two performance measures (time
mance measures depend on the kind of performed
and accuracy) into account (50). The MISTELS
exercise (e.g., suturing, cutting, clip applying) (45).
score, however, ignores a lot of information about
Therefore, different motion characteristics might be
instrument movement (e.g. the volume of move-
optimal for different exercises.
ment), which is an important issue in MIS. Therefore,
Using performance measures as an examination (or
the MISTELS score, although reliable and validated
classication) requires establishing a passing score
(51-53), can be seen as a score that only partially
(18). Determining a sufcient score is difcult
assesses actual psychomotor MIS skills.
because of (at least) two reasons:
Cotin et al. have introduced a standardised score
. Current assessment is done using multiple assess- based on ve motion analysis performance measures
ment measures, and (12). The score is supposed to be used to classify
. there is no gold standard of surgical competency residents according to their psychomotor MIS skills.
against which one can judge the validity of com- The score is calculated using user-dened weighting
petency assessment (18). factors. One of the shortcomings of that score is the
Objective classication 5

fact that the weight factors have not been reported. various tasks (e.g. suturing, placement of beads at
They are, thus, not standardised. It is, therefore, likely predened positions, and cutting predened pat-
that correct implementation of that score in the daily terns), and, since it is a fully objective method,
practice is difcult and can provide some confusion . there is no subjective bias introduced by the user or
when comparing scores of residents following differ- assessor.
ent MIS trainings. Moreover, the classication is
A pilot study showed the potential of this classi-
performed based on the training data of experts
cation method to distinguish between residents
only and does not account for the distribution of
with ne gradation in experience (e.g. experts and
performance measures of intermediates and/or
intermediates) (40). The classication method has
novices (12). Additionally, validation of the score
not been fully validated, and as such, it should not
included comparison of only expert and novice per-
(yet) be used to certify psychomotor MIS skills.
formances. In consequence, it is not known whether
Nevertheless, the potential of that classication
the standardised score is able to detect ne gradations
method lies in its simplicity, generalisability, and
in basic MIS skills.
objectiveness.
A fully objective classication method to differen-
tiate between surgically competent and surgically not
(yet) competent residents based on psychomotor MIS
skills has been introduced by Chmarra et al. (40). Factors that inuence psychomotor MIS skills
This method makes use of principal component
In the literature, the potential of using motion analysis
analysis (PCA) a data reduction technique, and
as an objective tool for assessing psychomotor MIS
linear discriminant analysis (LDA) a classication
skills in training environments has been demonstrated
method. Both PCA and LDA are commonly used in
(35-39). However, extensive research should take
the eld of pattern recognition and can easily be
place in order to allow full and correct exploitation
implemented for classication of residents according
of motion analysis as an assessment tool.
to their psychomotor MIS skills (54, 55). Among
MIS requires psychomotor skills different from
others, the most important advantages of using
open surgery. At Delft University of Technology,
such a classication method are:
three aspects (factors) that differentiate MIS from
. Possibility of using any motion analysis perfor- conventional open surgery have been investigated:
mance measure and/or combination of such Force feedback (56), use of the camera to obtain
measures, the view of the operation eld (45), and specic
. possibility of applying the method to a single spe- movements of MIS instrument obtained due to the
cic task (e.g., suturing) and to a combination of limited number of degrees of freedom (57). The

A. B.

4th DOF
3rd DOF

1st DOF

2nd DOF

Figure 2. The TrEndo tracking system developed at Delft University of Technology (58). The TrEndo is used for guiding and measuring
movements of real MIS instruments in training setups (box trainers and VR trainers). (A) A schematic drawing of the TrEndo. The TrEndo
allows measurement and manipulation of the instrument in four degrees of freedom (DOFs): translation (1st DOF) and rotation (2nd DOF) of
the instrument around its axis, and left-right (3rd DOF) and forward-backward (4th DOF) rotation of the instrument around the incision point.
(B) A box trainer equipped with the TrEndo tracking system.
6 M. K. Chmarra et al.

inuence of those three factors on MIS instrument movement in MIS is not performed via the shortest
movements has been studied. During those studies, path, and can be split into two phases: The retracting
the TrEndo tracking system (Delft University of phase and the seeking phase (Figure 4) (57). Only in
Technology, Figure 2) has been used to record move- the seeking phase, experts and intermediates are more
ments of real MIS instrument in the box trainer (58). effective than the novices. Therefore, the seeking
It has been found that force feedback (factor 1) phase is characteristic of performance differences
inuences psychomotor MIS skills when executed (57). Moreover, analysis of motions in separate phases
tasks require applying pulling and pushing forces of the movement gives great insight into the nature of
(56). For such tasks, a change from a trainer without the difference in performance of MIS tasks between
force feedback to the one that provides natural force expert and, e.g., novice performance. Therefore,
feedback has a detrimental effect on performance (e.g. motion analysis can be used to give feedback on
time, path length, depth perception). For that reason, the nature of possible residents limitations, which
it has been suggested that training for tasks in which have to be improved.
forces play an important role (e.g. stretching tissue)
should be done using trainers with natural force
feedback. In case of tasks that do not require appli- Discussion
cation of above mentioned forces (e.g. eye-hand
coordination tasks), training can be done in trainers Medical education extends over the lifetime of the
without force feedback. surgeon. Therefore, objective assessment methods are
In MIS, a small camera (factor 2) is used to provide needed to support regular certication and monitor-
the surgeon with the view of the operation eld. The ing of the residents progress, re-certication after a
view of the operation eld (e.g. side, angle, scale) can loss of privileges, and maintenance of certication
change when there is a change in the position of the (59). Although even a simple task in MIS requires
camera. This kind of change can disturb the eye-hand the use of different skills at the same time (e.g.
coordination of the surgeon (even an experienced knowledge of anatomy, eye-hand coordination, pro-
one), especially when combined with replacement tocol knowledge) (15, 16), only objective methods to
of the camera operator (as in Figure 3). Therefore, assess and to classify residents based on their psycho-
surgeons should be able to perform MIS well in two motor MIS skills have been presented in this paper.
situations: The idea behind that choice lies in the fact that
.
technical prociency seems to be fundamental to
When manipulating the camera themselves, and
.
perform surgery safely (18). This is particularly visible
when the assistant manipulates the camera.
in MIS, which is executed in a limited working area,
A study showed that camera holding (by the resi- with limited tactile perception, and with difcult
dent him/herself or by an assistant) inuences psy- handling of MIS instruments (60).
chomotor MIS skills (45). Particularly, this has been Although various methods to assess psychomotor
observable for novice residents. From the clinical MIS skills have been developed, none of them has
point of view, it was an interesting nding, because come out as clearly superior. Each of the methods has
it showed that residents who are hardly experienced in some advantages over the other and vice versa. The
MIS should begin their active participation in MIS in same can be said about methods that are used to
the OR by performing basic techniques while holding classify residents according to their psychomotor MIS
the camera themselves. skills.
Studies on specic movements of MIS instruments Motion analysis has a great potential to be used as a
obtained due to the limited number of degrees of means to objectively assess psychomotor MIS skills.
freedom (factor 3) showed that a goal-oriented For example, the study described by Chmarra et al.

M A M A M

P P
P

Figure 3. A schematic drawing representing three possible test congurations. Left: participant performing a task with the right hand,
and holding the camera in the left hand. Middle: participant performing a task with the right hand, the camera is held by an assistant.
Right: participant performing a task with the left hand, the camera is held by an assistant.
Objective classication 7

P
M

A
B
plane AB

plane ABP

Expert Novice
60 60
y [mm]

y [mm]
40 M M
40
20 20 plane ABP
0 A B 0 A B
0 20 40 60 80 100 120 0 20 40 60 80 100 120
x [mm] x [mm]
0 A B 0 A B
z [mm]

z [mm]

-20 -20 plane AB


-40 -40 M
M
-60 -60
0 20 40 60 80 100 120 0 20 40 60 80 100 120
x [mm] x [mm]

Figure 4. An example of goal-oriented movements of an MIS instrument due to limited number of degrees of freedom in MIS. Top: Plane
ABP (described by points A, B, and P) and plane AB (which passes through points A and B, and is perpendicular to plane ABP), with goal-
oriented movement (AMB) presented in three-dimensional space. Bottom: The typical trajectories for an expert (left) and a novice (right)
performing a goal-oriented movement. The movements are projected in the ABP and the AB planes. A the initial position of the tip of the
instrument; B the end position of the tip of the instrument; M a point, which is used to make a distinction between the retracting and seeking
phase; P incision (pivoting) point.

(57) shows that the thorough analysis of instrument instruments are used). In case of VR trainers, how-
movements in MIS can be used to indicate the size ever, the environment is less realistic (virtual) and the
and the nature of the gap between experts and novice MIS instruments are often simulated. Since not many
performance (12). In order to be able to fully and studies have been done on actual forces applied and
correctly exploit motion analysis as an assessment felt during MIS (61), force feedback in VR trainers (if
tool, more research still needs to be done. Moreover, present) can be very different from that provided by
to apply motion analysis as an assessment tool, it is real instruments. Nevertheless, the advantage of VR
necessary to equip trainers with a system that tracks trainers over other training methods is that due to the
and records these motions. Ideally, such a tracking use of simulated MIS instruments, the design of the
system should be applied at all stages of training (in tracking systems for a VR trainer is simplied. More-
the box trainer and VR trainer (during training), and over, in all VR trainers, tracking systems are inher-
in the OR (during an operation)) to provide the ently present (62). The systems that are able to track
resident with automatic detection of correctness of real MIS instruments, however, are still in their
performed task (a feature currently available only in infancy, since, e.g., factors such as patient safety,
VR trainers), and to allow to monitor his/her progress. size of the system, and ergonomics in the OR play
It could also be used to give feedback on the nature of a critical role in designing those systems. Ideally,
possible limitations of a resident. tracking systems should be designed so that it is
The OR is the most realistic (training) environ- possible to use them in all training environments.
ment, in which residents use real MIS instruments, Currently, however, there are only a few systems
and where realistic force feedback is obtained due to that can track movements of real MIS instruments
natural instrument-tissue interactions. Similarly, box in box trainers or in the OR (64). Most of those
trainers also provide residents with a realistic envi- systems are still at a prototype or a testing stage.
ronment and force feedback (because real MIS The systems available on the market, on the other
8 M. K. Chmarra et al.

hand, are very expensive. As a consequence, systems standardised and valid tasks and setups, which are
that track real instruments are mostly used in aca- described and dened in detail. Currently, in the
demic environments for research purposes (e.g., to literature on MIS, it is hardly possible to nd a
develop new assessment methods) (12, 38, 39, publication about an assessment method (or valida-
57, 63, 64). tion study) that reports each detail of the setup (e.g.
Assessing residents psychomotor MIS skills in position of the monitor, the operator of the camera,
almost realistic performance situations often results position of the task in a box trainer) and the protocol
in complex and not straightforward test design (e.g., used in the study. Without such information, it is
simulation of anatomy or bleeding in VR trainers). difcult to reproduce the tests or to implement
Besides that, independent of the reality level in a described assessment methods.
trainer, it is still a simulation. For that reason, resi- Lack of knowledge on all the factors that have an
dents do not behave exactly in the same way as they inuence on psychomotor MIS skills shows that there
would in a real-life surgery. Consequently, assessment is a need to identify those factors. For example, it is
of psychomotor MIS skills can easily be biased. still not known what kinds of forces are felt and
In the literature, identication (and quantication) should be applied when performing MIS. Moreover,
of factors that inuence psychomotor MIS skills have it has not yet been recognised whether the same forces
not yet gained much attention. Nevertheless, when are (and should be) applied during training of psy-
developing and/or introducing assessment and/or chomotor MIS skills. Does the camera holding (by an
classication methods, it is necessary to take into assistant or by the participant him/herself) inuence
account that those factors can inuence (bias) the forces applied during performing MIS tasks? When
outcomes of those methods. There are a few impor- should application of forces be introduced into the
tant lessons learnt from studies on identifying such training curriculum? How exactly are psychomotor
factors. Lesson 1: There are factors that inuence MIS skills inuenced by the presence or absence of
motions of MIS instruments. Ignoring those factors force feedback? Can we measure it? Which motion
when assessing psychomotor MIS skills will bias that analysis performance measures can be used to deter-
assessment. Lesson 2: The judgement on psychomo- mine that? These and many more other questions
tor MIS skills of residents who perform tasks that need to be answered in future research. Only then will
require applying pulling and pushing forces (e.g. one be able to develop and implement standard,
suturing) in a trainer without correct force feedback correct and objective assessment and classication
needs to be done carefully, since there is a high methods.
possibility that motions of MIS instrument(s) are Classication methods are supposed to provide an
different than those during executing exactly the aid in deciding whether a resident is competent to
same task in a trainer with natural force feedback. move to the next level of training (e.g. to perform an
Lesson 3: The setting (e.g. task, position of the operation in the OR) as well as to certify and monitor
camera, camera operator, MIS instrument) used progress of residents. Objective assessment methods,
when assessing psychomotor MIS skills inuences on the other hand, can be used to identify possible
those skills. Moreover, the setting plays an important residents limitations, which have to be improved.
role when determining parameters to discriminate There is, therefore, a place for both classication
between different levels of performance. Lesson 4: and assessment methods in MIS. Motion analysis
The concept of shortest path length, which is currently has a potential to become the means to objectively
used to indicate better MIS performance, may not be assess psychomotor MIS skills. It can be expected that
a proper concept for analysing optimal movements. simple, general, and objective classication methods
Since no studies have been done to nd out which (e.g. such as the method based on PCA and LDA) will
motion analysis performance measures truly can be used in the future to determine whether a
quantify psychomotor MIS skills, it is not sure residents skills level can be called expert,
whether all of the currently used measures are worthy intermediate, or novice.
to be used. As mentioned in the Introduction, ideally, objective
As said above, there are factors that inuence MIS assessment methods address at least two topics: They
instruments movements when performing MIS task identify potential areas of improvement, and they
(e.g. presence/absence of force feedback, setup used determine whether a resident is competent to move
for assessment, fatigue and sleep deprivation (65)). to the next level of training (e.g., to the OR). Iden-
For that reason, it is necessary to take those factors tication of potential areas of improvement of skills
into account when developing assessment and/or can be done by thorough analysis of instrument
classication methods. Therefore, it is recommended motion such as, e.g., in case of study on specic
to evaluate residents MIS skills only using movements of MIS instrument obtained due to the
Objective classication 9

limited number of degrees of freedom (factor 3) (58). experts, intermediates, and novices based on their
Thus far, however, there are only a few studies that psychomotor skills.
investigated movements in MIS in detail.
Delcaration of interest: The authors have no con-
To determine whether a resident is competent to
icts of interest or nancial ties to disclose.
move to the next level of training, a passing score
needs to be established. As said in the Classication
based on basic psychomotor MIS skills section, References
determining a sufcient score is difcult because
current assessment is done using multiple assessment 1. Park AE, Witzke D. The surgical competence conundrum.
measures, and because there is no gold standard of Surg Endosc. 2002;16:5557.
surgical competency against which one can judge the 2. Roberts KE, Bell RL, Duffy AJ. Evolution of surgical skills
training. World J Gastroenterol. 2006;12:321924.
validity of competency assessment. Therefore, before 3. Ritchie WP. Basic certication in surgery by the American
the passing score can be determined, it has to be Board of Surgery (ABS). What does it mean? Does it have
decided which assessment measures are going to be value? Is it relevant? A personal opinion. Ann Surg.
used, and the gold standard of surgical competency 2004;239:1339.
needs to be established. 4. Halvorsen FH, Elle OJ, Fosse E. Simulators in surgery. Minim
Invasiv Ther. & Allied Technol. 2005;14:21423.
Independent of the kind of method used to assess 5. Aggarwal R, Moorthy K, Darzi A. Laparoscopic skills training
MIS skills, performance in one context (e.g. psycho- and assessment. Br J Surg. 2004;91:154958.
motor MIS skills) does not predict performance in 6. Gallagher AG, Lederman AB, McGlade K, Satava RM,
another context (e.g. knowledge of anatomy) very Smith CD. Discriminative validity of the minimally invasive
well. Therefore, motion analysis alone is not a surgical trainer in virtual reality (MISTVR) using criteria
levels based on expert performance. Surg Endosc.
panacea, but a method that is not capable of dem- 2004;18:6605.
onstrating the actual surgical competence of the indi- 7. McClusky DAM, van Sickle K, Gallagher AG. Relationship
vidual. Therefore, it is required to make a selection of between motion analysis, time, accuracy, and errors during
assessment methods that depends on the skills to be performance of a laparoscopic suturing task on an augmented
evaluated, and to use a combination of those reality simulator. 12th International Congress of the European
Association for Endoscopic Surgery, Barcelona, O207,
methods. June 2004.
Any fully objective assessment of MIS skills is more 8. Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A. Laparo-
reliable than a subjective one. When using objective scopic virtual reality and box trainers: is one superior to the
motion analysis methods to assess psychomotor MIS other? Surg Endosc. 2004;18:48594.
skills it is more likely that residents will accept feed- 9. Schijven M, Jakimowicz J. Virtual reality surgical laparoscopic
simulators. How to choose. Surg Endosc. 2003;17:194350.
back on their skills and that they will constructively 10. Sutherland LM, Middleton PF, Anthony A, Hamdorf J,
incorporate that knowledge in training. Cregan P, Scott D, et al. Surgical simulation. A systematic
review. Ann Surg. 2006;243:291300.
11. Darzi A, Datta V, MacKay S. The challenge of objective
assessment of surgical skill. Am J Surg. 2001;181:4846.
Conclusions 12. Cotin S, Stylopoulos N, Ottensmeyer M, Neumann P,
Rattner D, Dawson S. Metrics for Laparoscopic Skills
Currently used objective methods to assess and clas- Trainers: The Weakest Link! Lect Notes Comput Sc
sify residents according to their basic psychomotor LNCS 2002;2488:3543.
MIS skills are based on motion analysis. These meth- 13. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J,
ods are supposed to address at least two topics: Hutchison C, et al. Objective structured assessment of tech-
nical skills (OSATS) for surgical residents. Brit J Surg.
Identifying potential areas of improvement of skills,
1997;84:2738.
and determining whether a resident is competent to 14. Moorthy K, Munz Y, Dossis A, Bello F, Darzi A. Motion analysis
move to the next level of training (e.g., to the OR). in the training and assessment of minimally invasive surgery.
Motion analysis has a potential to be the means to deal Minim Invasiv Ther & Allied Technol. 2003;12:13742.
with those two topics. However, current lack of gen- 15. Dankelman J, Chmarra MK, Verdaasdonk EGG,
Stassen LPS, Grimbergen CA. Fundamental aspects of learn-
eral knowledge on basic psychomotor skills in MIS
ing minimally invasive surgical skills. Minim Invasiv Ther &
does not (yet) enable to fully and correctly exploit Allied Technol. 2005; 14:24756.
motion analysis as an assessment tool, e.g. for certi- 16. Cuschieri A, Francis SN, Crosby J, Hanna GB. What do
cation purposes (e.g. assessment of psychomotor master surgeons think of surgical competence and revalida-
skills of residents before they receive their specialty tion? Am J Surg. 2001;182:1106.
17. Park AE, Witzke D. The surgical competence conundrum.
certication). Nevertheless, motion analysis together
Surg Endosc. 2002;16:5557.
with simple, general, and objective classication 18. Feldman LS, Sherman V, Fried GM. Using simulators to
methods (e.g. such as linear discriminant analysis) assess laparoscopic competence: ready for widespread use?
can already be used to objectively classify residents as Surgery 2004;135:2842.
10 M. K. Chmarra et al.

19. Moore MJ, Bennett CL. The learning curve for laparoscopic 38. Strom P, Kjelling A, Hedman L, Wredmark T, Fellander-
cholecystectomy. Am J Surg. 1995;170:5509. Tsai L. Training in tasks with different visual-spatial compo-
20. Morgenstern L, McGrath MF, Carroll BJ, Paz-Partlow M, nents does not improve arthroscopy performance. Surg
Berci G. Continuing hazards of the learning curve in laparo- Endosc. 2004;18:11520.
scopic cholecystectomy. Am Surg. 1995;61:9148. 39. Smith CD, Farrell TM, McNatt SS, Metreveli RE. Assessing
21. Lekawa M, Shapiro SJ, Gordon LA, Rothbart J, Hiatt JR. The laparoscopic manipulative skills. Am J Surg. 2001;181:54750.
laparoscopic learning curve. Surg Laparosc Endosc. 40. Chmarra MK, Klein S, de Winter JCF, Jansen FW,
1995;5:4558. Dankelman J. Objective classication of residents based on
22. Mehrabi A, Yetimoglu CL, Nickkholgh A, Kash A, Kienle P, their psychomotor laparoscopic skills. Surg Endoscop, 2009.
Konstantinides L, et al. Development and evaluation of a Epub ahead of print.
training module for the clinical introduction of the da Vinci 41. Cesanek P, Uchal M, Uranues S, Patruno J, Gogal C,
robotic system in visceral and vascular surgery. Surg Endosc. Kimmel S, Bergamaschi R. Do hybrid simulator-generated
2006;20:137682. metrics correlate with content-valid outcome measures? Surg
23. Passerotti CC, Nguyen HT, Retik AB, Peters CA. Patterns Endosc. 2008;22:217883.
and Predictors of Laparoscopic Complications in Pediatric 42. Aggarwal R, Grantcharov T, Moorthy K, Milland T,
Urology: The Role of Ongoing Surgical Volume and Access Papasavas P, Dosis A, et al. An evaluation of the feasibility,
Techniques. J Urology 2008;180:6815. validity, and reliability of laparoscopic skills assessment in the
24. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, operating room. Ann Surg. 2007;245:9929.
Leffondre K, Stanbridge D, et al. A global assessment tool 43. Moorthy K, Munz Y, Dosis A, Bello F, Chang A, Darzi A.
for evaluation of intraoperative laparoscopic skills. Am J Surg. Bimodal assessment of laparoscopic suturing skills. Construct
2005;190:10713. and concurrent validity. Surg Endosc. 2004;18:160812.
25. Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident 44. Stefanidis D, Haluck R, Pham T, Dunne JB, Reinke T,
laparoscopic performance using global operative assessment of Markley S, et al. Construct and face validity and task workload
laparoscopic skills. J Am Coll Surg. 2007;204:30813. for laparoscopic camera navigation: virtual reality versus
26. Chang L, Hogle NJ, Moore BB, Graham MJ, Sinanan MN, videotrainer systems at the SAGES Learning Center. Surg
Bailey R, et al. Reliable assessment of laparoscopic perfor- Endosc. 2007;21:115864.
mance in the operating room using videotape analysis. Surg 45. Chmarra MK, Kolkman W, Jansen FW, Grimbergen CA,
Innov. 2007;14:1226. Dankelman J. The inuence of experience and camera holding
27. Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A. on laparoscopic instrument movements measured with the
Toward feasible, valid, and reliable video-based assessments TrEndo tracking system. Surg Endosc. 2007;21:206975.
of technical surgical skills in the operating room. Ann Surg. 46. Darzi A, Datta V, Mackay S. The challenge of objective
2008;247:3729. assessment of surgical skills. Am J Surg. 2001;181:4846.
28. Pellen M, Horgan L, Barton JR, Attwood S. Laparoscopic 47. Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF,
surgical skills assessment: can simulators replace experts? Schirmer B, et al. Development and validation of a compre-
World J Surg. 2010;33:4407. hensive program of education and assessment of the basic
29. McKinley RK, Strand J, Ward L, Gray T, Alun-Jones T, fundamentals of laparoscopic surgery. Surgery 2004;135:
Miller H. Checklists for assessment and certication of clinical 2127.
procedural skills omit essential competencies: a systematic 48. Fundamentals of Laparoscopic Surgery (FLS) Program. Soci-
review. Med Educ. 2008;42:33849. ety of American Gastrointestinal and Endoscopic Surgeons.
30. Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assess- http://www.sprogram.org
ment of technical skills in surgery. BMJ 2003;327:10327. 49. Ritter EM, Scott DJ. Design of a Prociency-Based Skills
31. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing Training Curriculum for the Fundamentals of Laparoscopic
competency in surgery: where to begin? Surgery Surgery. Surg Innov. 2007;14:10712.
2004;135:620. 50. Vassiliou MC, Ghitulescu GA, Feldman LS, Stanbridge D,
32. Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS. Devel- Leffondre K, Sigman HH, et al. The MISTELS program to
opment of an objective structured assessment of technical measure technical skill in laparoscopic surgery : evidence for
skills for obstetric and gynecology residents. Obstet Gynecol. reliability. Surg Endosc. 2006;20:7447.
2000;96:14650. 51. Scott DJ, Ritter EM, Tesfay ST, Pimentel EA, Nagji A,
33. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Fried GM. Certication pass rate of 100% for fundamentals
Testing technical skill via an innovative bench station of laparoscopic surgery skills after prociency-based training.
examination. Am J Surg. 1997;173:22630. Surg Endosc. 2008;22:188793.
34. Dosis A, Aggarwal R, Bello F, Moorthy K, Munz Y, Gillies D, 52. Fried GM. FLS assessment of competency using simulated
et al. Synchronized video and motion analysis for the assess- laparoscopic tasks. J Gastrointest Surg. 2008;12:2102.
ment of procedures in the operating theater. Arch Surg. 53. McCluney AL, Vassiliou MC, Kaneva PA, Cao J,
2005;140:2939. Stanbridge DD, Feldman LS, et al. FLS simulator perfor-
35. van Sickle KR, McClusky DA, Gallagher AG, Smith CD. mance predicts intraoperative laparoscopic skill. Surg Endosc.
Construct validation of the ProMIS simulator using a novel 2007;21:19915.
laparoscopic suturing task. Surg Endosc. 2005;19:122731. 54. Jain AK, Duin RPW, Mao J. Statistical pattern recognition: a
36. Acosta E, Temkin B. Haptic laparoscopic skills trainer with review. IEEE Trans Pattern Anal Machine Intell. 2000;
practical user evaluation metrics. Stud Health Techn Inform, 22:438.
MMVR 2005;111:8-11. 55. Duda RO, Hart PE, Stork DG. Pattern classication. Second
37. Cavallo F, Megali G, Sinigaglia S, Tonet O, Dario P. A ed. Willey, New York, 2001.
biomechanical analysis of surgeons gesture in a laparoscopic 56. Chmarra MK, Dankelman J, van den Dobbelsteen JJ,
virtual scenario. Stud Health Technol Inform, MMVR Jansen FW. Force feedback and basic laparoscopic skills.
2006;14:7984. Surg Endosc. 2008;22:21408.
Objective classication 11

57. Chmarra MK, Jansen FW, Grimbergen CA, Dankelman J. feedback in grasping tissue laparoscopically. Surg Endosc.
Retracting and Seeking Movements During Laparoscopic 2004;18:9805.
Goal-Oriented Movements. Is the Shortest Path Length 62. Chmarra MK, Grimbergen CA, Dankelman J.
Optimal? Surg Endosc. 2008;22:9439. Systems for tracking minimally invasive surgical instru-
58. Chmarra MK, Bakker NH, Grimbergen CA, Dankelman J. ments. Minim Invasiv Ther & Allied Technol. 2007;16:
TrEndo, a device for tracking minimally invasive surgical 32840.
instruments in training set-ups. Sensor Actuat A-Phys. 63. Sokollik C, Gross J, Buess G. New model for skills assessment
2006;126:32834. and training progress in minimally invasive surgery. Surg
59. Satava RM. Assessing surgery skills through simulation. Endosc. 2004;18:495500.
Clinical Teacher 2006;3:10711. 64. Rosen J, Brown JD, Chang L, Sinanan M, Hannaford B.
60. Lehmann KS, Ritz JP, Maass H, Cakmak HK, Generalized approach for modeling minimally invasive
Keuhnapfel UG, Germer CT, et al. A prospective randomized surgery as a stochastic process using a discrete Markov
study to test the transfer of basic psychomotor skills from model. IEEE Transactions on Biomedical Engineering
virtual reality to physical reality in a comparable training 2006;53:399413.
setting. Ann Surg. 2005;241:4429. 65. Kahol K, Leyba MJ, Deka M, Deka V, Mayes S, et al. Effect of
61. Heijnsdijk EAM, Pasdeloup A, van der Pijl AJ, Dankelman J, fatigue on psychomotor and cognitive skills. Am J Surg.
Gouma DJ. The inuence of force feedback and visual 2008;195:195204.
Copyright of Minimally Invasive Therapy & Allied Technologies is the property of Taylor & Francis Ltd and
its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for individual use.