You are on page 1of 5

Surg Endosc (2004) 18: 16081612

DOI: 10.1007/s00464-003-9312-5

Springer Science+Business Media, Inc. 2004

Bimodal assessment of laparoscopic suturing skills


Construct and concurrent validity
K. Moorthy, Y. Munz, A. Dosis, F. Bello, A. Chang, A. Darzi
Department of Surgical Oncology and Technology, Imperial College, St. Marys Hospital, 10th Floor, QEQM Building, Praed Street,
W2 1NY, London

Received: 27 November 2003/Accepted: 27 May 2004/Online publication: 13 October 2004

Abstract The high incidence of complications associated with


Background: The assessment of technical skills should laparoscopic cholecystectomy during the early 1990s [4]
provide objective feedback and judge suitability of was probably a result of the failure to appreciate the fact
progress during training. The aim of this study was to that the performance of laparoscopic surgery required
validate two objective assessment techniques for lapa- its own set of unique skills compared to open surgery. In
roscopic suturing and demonstrate a correlation be- laparoscopic surgery, surgeons have only a two-dimen-
tween them. sional view of the operative eld, there is an absence of
Methods: Six experts, seven surgeons skilled in laparo- tactile feedback, they have to get accustomed to the
scopic suturing, and 13 with no laparoscopic suturing fulcrum eect, and they also have to develop intuitive
skills were asked to place two or three intracorporeal handeye coordination and depth perception [7, 14, 18].
sutures on a synthetic suture pad. The latter group was As a result, there is a learning curve associated with
given video-based instructions prior to the execution of laparoscopic procedures. It has also been found that
the sutures. Ergonomic conditions were standardized for learning curves vary for dierent procedures. For
all subjects. The procedures were recorded on videotape example, the learning curve for laparoscopic cholecys-
and two blinded observers rated the rst suture of each tectomy is approximately 2550 procedures [1, 9] and
subject on a 29-point checklist. A motion analysis sys- for laparoscopic colorectal surgery approximately 50
tem, Imperial College Surgical Assessment Device, was procedures [6]. However, even the number of procedures
used to assess psychomotor skills. required to achieve competence varies among studies [6,
Results: There was a signicant dierence in the time 17]. In addition, learning curves for laparoscopic surgery
taken (p = 0.000) and total path length (p = 0.000) per usually rely on time taken for the procedure [11]. Time is
suture across the groups. There were also a signicant a crude measure because it depends not only on the skill
dierence in the total checklist score (p = 0.000) and its of the surgeon but also on patient variability. Therefore,
individual categories. The was a strong correlation be- if surgeons have to receive accreditation for procedures,
tween the total path length and the total checklist score as has been suggested [16], the process of assessment
(coecient, 0.78; p < 0.001). should be made more objective and reliable.
Conclusions: A combination of the motion analysis One of the main drawbacks to the current model of
system and the checklist would make the process of psychomotor skills learning is the absence of objective
formative feedback during the learning of intracorporeal feedback [13]. Feedback is essential because deciencies
suturing objective and comprehensive. in training are dicult to correct without feedback [8].
In addition, feedback should be as objective as possible
Key words: Technical skills and surgery Laparo- because surgical trainees already have a certain percep-
scopic skills and assessment Motion analysis tion of self that may be dicult to inuence. Even
though technical prociency is a small aspect of surgical
competence [20], it is considered to be of crucial
importance especially during training [3]. In a workshop
of master surgeons, the consensus was that technical
skills assessment should be an important part of a sur-
gical training program [3]. However, the assessment of
skills depends largely on the subjective evaluation of
Correspondence to: K. Moorthy trainers. It has been suggested that assessment by direct
1609

observation without criteria has only a moderate level of


validity and little reliability because it is often dicult
for two trainers to agree on the skill level of a trainee
[13].
In 1971, Kopta [12] suggested the use of checklists
for the assessment of technical skills among orthopedic
trainees. Since then, checklists have been used across a
number of specialties, including laparoscopic surgery [5].
The availability of set criteria against which technical
skills can be assessed makes the assessment process more
objective, valid, and reliable [22]. It has been said that
checklists turn examiners into observers of behavior
rather than interpreters of behavior, thereby removing
the subjectivity of the evaluation process [12].
The aim of this study was to develop and establish
the construct validity of a checklist for the assessment of
laparoscopic suturing and to establish its concurrent
validity against a validated method of assessment of Fig. 1. Imperial College Surgical Assessment Device. A, signal gen-
erator; B, receivers/sensors.
technical skilldexterity analysis.

Materials and methods

Surgeons of varying levels of experience in laparoscopic surgery were


enrolled in the study. Informed consent was obtained from all par-
ticipants. The surgeons were divided into three groups. Group 1 con-
sisted of experts who had performed more than 100 endoscopic sutures
during operative procedures or had frequently practiced in pelvi-
trainers; group 2 consisted of surgeons who possessed laparoscopic
skills from having performed 1050 laparoscopic cholecystectomies
and had performed less than 50 endoscopic sutures in a pelvi-trainer;
group 3 consisted of trainees who had performed fewer than 10 lap-
aroscopic cholecystectomies but did not possess any laparoscopic
suturing skills.

Study design
All the surgeons were asked to place between one and three sutures on
a synthetic suture pad (Annexe Art, Anglesey, Wales, UK) placed
within a box trainer within a period of 15 minutes. There were pre-
determined marks on the suture pad on either side of an incision
through which the participants were instructed that the needle should
enter and exit. They were instructed to exit through the incision before
going through the opposite side. The angle of the laparoscope, the
distance of the laparoscope from the suture pad, the port positions, the
suture material (30 vicryl, USSC, Norwalk, CT, USA), and the needle
holders were all standardized. All surgeons were requested to under-
take the two-handed collapsed reef knot technique followed by a single
locking throw as taught in the endoscopic suturing course conducted
by the Royal College of Surgeons of England. The trainees from group
3 were given video-based instruction on endoscopic suturing by an Fig. 2. Check list for endoscopic suturing.
expert surgeon prior to performing the task.

In addition, all the tasks were videotaped, and the rst suture of
Assessment of performance each participant was played back to two independent blinded
observers who rated the task performance using the checklist.
Imperial college surgical assessment device
This is an electromagnetic motion tracking system in which the posi- Checklist
tional data from sensors worn on the dorsum of a surgeons hands are
converted into meaningful dexterity data using special software The checklist consists of six categories of 29 items, which are scored as
(Fig. 1). The dexterity parameters are the number of movements, the a 1 or 0 (Fig. 2). These categories are needle loading (NL), needle
path length traveled by the hands, the speed of movements, and the driving (ND), pulling the suture through (PS), technique of tying knots
time taken for the task. Previous studies have shown that path length is (knot technique, KT), Knot slippage (KS), and knot quality (KQ). The
a discriminator of skill (construct validity) for simple [21] and complex rst two categories are presented as two phases because the subjects
[19] laparoscopic tasks. Thus, time and path length per suture were were requested to exit though the incision prior to driving the needle
used as the dexterity parameters for this study. through the opposite side. For purposes of standardization, the par-
1610

Table 1. Assessment parametersa

Time in Path length Checklist score


Group No. (sec) (cm) as % age

1 (n = 6) 128 (37.2) 1299.23 (437.7) 84.46 (9.8)


2 (n = 7) 248 (131.8) 2110.04 (708.9) 63.6 (20.05)
3 (n = 13) 667 (432.2) 10050 (7554.5) 28.8 (19.2)
a
Median values; values in parentheses, interquartile range

Fig. 4. Path length per suture in centimeters. The central black line is
the median, the data within the box are the interquartile range, and the
ends of the vertical line denotine the whole range. The circle denotes
outliers.

Fig. 3. Time taken per suture in seconds. The central black line is the
median, the data within the box are the interquartile range, and the
ends of the vertical line denote the whole range.

ticipants were requested to load the needle on the needle holder and
place the instruments on the suture pad before the Imperial College
Surgical Assessment Device (ICSAD) was started, and they were then
asked to commence the task. As result, two of the items on the
checklist relating to needle positioning before the task commenced
were eliminated from the analysis. The total checklist score is expressed
as a percentage. Comparison between the groups consisted of ana-
lyzing the total checklist score as well as the six individual categories.

Data analysis
Fig. 5. Checklist score for rst suture; % age, percentage score. The
Nonparametric tests were used for statistical analyses. Construct central black line is the median, the data within the box are the in-
validity was established by analyzing dierences across all groups terquartile range, and the ends of the vertical line denote the whole
using the Kruskal-Wallis test. The Mann-Whitney U test was used to range. The circle denote outliers.
analyze the dierences between the groups. We also performed an
intergroup analysis using the Mann-Whitney U test to evaluate any
dierences between the subjects in group 3 who possessed laparoscopic (Fig. 3), for the path length per suture (p = 0.000)
skills and those who did not.
Spearmans rank correlation test was used to demonstrate the (Fig. 4), and for the total checklist score (p = 0.000)
correlation between the path length and the checklist score. We (Fig. 5). Analysis of the individual categories of the
determined the interrater reliability for the checklist score using checklist revealed that there was a signicant dierence
Cronbachs alpha coecient. across the groups for four of the six components except
PS and KS (Figs. 6 and 7).
There were signicant dierences between groups 1
Results and 2 for path length (p = 0.01) but not for time taken
per suture (p = 0.06). There were signicant dierences
There were six surgeons in group 1, seven in group 2, between groups 2 and 3 for time (p = 0.001) and for
and 13 in group 3. Table 1 gives the median and inter- path length (p = 0.000).
quartile ranges of all three groups for the assessment There was a signicant dierence between groups 1
parameters. There was a signicant dierence across the and 2 (p = 0.01) and between groups 2 and 3 for the
three groups for the time taken per suture (p = 0.000) total checklist score (p = 0.001). Comparison between
1611

the checklist score (rho = )0.78, p = 0.000). The in-


terrater reliability for the checklist score was 0.90.

Discussion

Time to perform the task has been the only objective


measure used for the assessment of endoscopic suturing
skills in many studies [2, 10, 15]. Smith et al. [18] stated
that simply measuring the operative time does not
indicate mastery of laparoscopic manipulative skills.
They also suggested that accuracy is a more sensitive
indicator of skill than time. In our study, although there
was a signicant dierence across all the groups, time
per suture was not sensitive enough to detect a dierence
between the two groups that possessed intracorporeal
endoscopic suturing skill. ICSADs parameters help in
Fig. 6. Cluster box plots for needle loading (NL) and needle driving quantifying dexterity and in understanding expert per-
(ND). The central black line is the median, the data within the box are formance. In a previous study, it was found that expert
the interquartile range, and the ends of the vertical line denote the
whole range. Circles denote outliers, and asterisks denote the extreme
surgeons perform a task with a lower path length, which
observations. reects their economy of motion and their accuracy [19,
21]. Similarly, in our study the path length per suture
dierentiated the skill level across all three groups of
surgeons and was sensitive enough to detect the dier-
ence between the two groups that possessed suturing
skills.
This study has demonstrated the construct validity
of a checklist developed for the assessment of endo-
scopic suturing skills. This checklist was developed after
discussions with expert surgeons. The six categories of
the checklist are similar to the categories of assessment
of suturing skills developed by Mori et al. [10]. Our
study has shown that the total checklist score is sensitive
to detect dierences across the groups and even in be-
tween the groups.
Checklists make the assessment of psychomotor
skills criteria based and thus increase the validity and
reliability of the assessment process [13]. In a previous
study comparing a checklist and a global rating scale
embedded within the Objective Structured Assessment
of Technical Skills, checklists were found to possess
Fig. 7. Cluster box plots for knot technique (KT) and knot quality
poor reliability in comparison to the global rating scale
(KQ). The central black line is the median, the data within the box are [12]. One of the reasons for the high interrater reliability
the interquartile range, and the ends of the vertical line denote the of our checklist is probably that every checklist item was
whole range. Circles denote outliers, and asterisks denote the extreme anchored by explicit criteria for that particular item to
observations. be scored as a 1 or a 0. For example, if there was more
than one entry through the suture pad, or even one
fumble during the throws while tying the knot, the item
was scored as a 0.
the groups for the individual checklist categories re- The checklist evaluates task execution for mainly
vealed that although group 1 scored higher than group 2 psychomotor skills and bimanual coordination. How-
(Figs. 6 and 7) for nearly all the checklist categories, the ever, there are some items that assess safety, such as the
dierence was only signicant for KT (p = 0.02). There needle on the needle holder being in view at all times and
was a signicant dierence between groups 2 and 3 for the knots being squared and secured. All categories of
all the categories except PS and KS (Figs. 6 and 7). the checklist, except for PS and KS, were able to detect
Subgroup analysis of group 3 revealed that there dierences across the groups. One of the reasons for the
were no dierences between the trainees who had pre- apparent lack of dierence for KS is that surgeons from
viously performed laparoscopic cholecystectomy and even group 1 (experts) did not keep the needle held on
those who had not (time, p = 0.38; path length, the needle loader in view while pulling the suture
p = 0.66; checklist, p = 0.92). There was also a strong through. This may be considered to be an unsafe act due
and signicant correlation between the path length and to the likelihood of inadvertent damage to nearby
1612

structures. The experts did score higher than the inter- 4. Deziel DJ, Millikan KW, Economou SG, Doolas A, Ko ST, Airan
mediate group on nearly all categories but the dierence MC (1993) Complications of laparoscopic cholecystectomy: a
national survey of 4,292 hospitals and an analysis of 77,604 cases.
was only signicant for KT. This is one of the most Am J Surg 165: 914
complex tasks in laparoscopic surgery and requires 5. Eubanks TR, Clements RH, Pohl D, Williams N, Schaad DC,
bimanual dexterity and probably is achieved only with Horgan S, et al. (1999) An objective scoring system for laparo-
considerable practice. scopic cholecystectomy. J Am Coll Surg 189: 566574
6. Fleshman JW, Fry RD, Birnbaum EH, Kodner IJ (1996) Lapa-
We have also demonstrated the concurrent validity roscopic-assisted and minilaparotomy approaches to colorectal
of the checklist by demonstrating the strong correlation diseases are similar in early outcome. Dis Colon Rectum 39: 1522
between the checklist score and path length. Despite 7. Gallagher AG, Satava RM (2002) Virtual reality as a metric for
this, checklists and motion analysis systems do have the assessment of laparoscopic psychomotor skills. Learning
drawbacks. Checklist assessments involve either direct curves and reliability measures. Surg Endosc 16: 17461752
8. Kopta JA (1971) An approach to the evaluation of operative skills.
observations of task performance or periods of intense Surgery 70: 297303
video watching. ICSAD is a quantitative measure of 9. Lekawa M, Shapiro SJ, Gordon LA, Rothbart J, Hiatt JR (1995)
skill and does not give any information about the The laparoscopic learning curve. Surg Laparosc Endosc 5: 455
quality of performance. For example, a surgeon could 458
10. Mori T, Hatano N, Maruyama S, Atomi Y (1998) Signicance of
execute a laparoscopic knot without squaring the hands-on training in laparoscopic surgery. Surg Endosc 12:
throws, take fewer throws, or complete a loose knot and 256260
yet execute the knot with fewer movements and a lower 11. Peters JH, Ellison EC, Innes JT, Liss JL, Nichols KE, Lomano
path length. Thus, a comprehensive assessment of per- JM, et al. (1991) Safety and ecacy of laparoscopic cholecystec-
formance should integrate quantitative measures of tomy. A prospective analysis of 100 initial patients. Ann Surg 213:
312
dexterity using ICSAD with a qualitative assessment of 12. Regehr G, MacRae H, Reznick RK, Szalay D (1998) Comparing
performance using checklists. The main aim of our the psychometric properties of checklists and global rating scales
assessment method of combining the checklist and IC- for assessing performance on an OSCE-format examination. Acad
SAD is to make training in endoscopic suturing skills Med 73: 993997
13. Reznick RK (1993) Teaching and testing technical skills. Am J
superior with objective feedback. Surg 165: 358361
Other than the obvious benets to training and 14. Rosser JC, Rosser LE, Savalgi RS (1997) Skill acquisition and
credentialing of surgeons, objective measures of lapa- assessment for laparoscopic surgery. Arch Surg 132: 200204
roscopic skills assessment may have extensive research 15. Rosser JC Jr, Rosser LE, Savalgi RS (1998) Objective evaluation
applications, such as the evaluation of the benets of of a laparoscopic surgical skill program for residents and senior
surgeons. Arch Surg 133: 657661
training with virtual reality systems, for studying the 16. Royston CM, Lansdown MR, Brough WA (1994) Teaching lap-
eect of adverse conditions such as sleep deprivation aroscopic surgery: the need for guidelines. Br Med J 308: 1023
and shift patterns on performance, and to objectively 1025
demonstrate the benet of training courses. 17. Simons AJ, Anthone GJ, Ortega AE, Franklin M, Fleshman J,
Geis WP, et al. (1995) Laparoscopic-assisted colectomy learning
curve. Dis Colon Rectum 38: 600603
18. Smith CD, Farrell TM, McNatt SS, Metreveli RE (2001) Assess-
ing laparoscopic manipulative skills. Am J Surg 181: 547550
References 19. Smith SG, Torkington J, Brown TJ, Tander NJ, Darzi A (2002)
Motion analysis. Surg Endosc 16: 640645
1. Cagir B, Rangraj M, Mauci L, Herz BL (1994) The learning 20. Spencer FC, Spencer FC (1978) Teaching and measuring surgical
curve for laparoscopic cholecystectomy. J Laparoendosc Surg 4: techniquesthe technical evaluation of competence. Bull Am Coll
419427 Surgeons 63: 912
2. Chung JY, Sackier JM (1998) A method of objectively evalu- 21. Tander N, Smith S, Mair J, Russell R, Darzi A (1999) Can a
ating improvements in laparoscopic skills. Surg Endosc 12: computer measure surgical precision? Reliability, validity and
11111116 feasibility of the ICSAD. Surg Endosc 13(suppl 1): 81
3. Cuschieri A, Francis N, Crosby J, Hanna GB (2001) What do 22. Watts J, Feldman WB (1985) Assessment of technical skills. In:
master surgeons think of surgical competence and revalidation? Neufeld VR, Norman GR (eds). Assessing clinical competence.
Am J Surg 182: 110116 Springer, New York, pp 259274