Beruflich Dokumente
Kultur Dokumente
Abstract
Purpose of the learner, such as their ability approaches were often combined
To identify approaches to operationalizing to perform different, increasingly within a single set of milestones
the development of competence in difficult tasks (304/601; 51%), (mean 3.9, SD 1.6).
Accreditation Council for Graduate perform a task better and faster
Medical Education (ACGME) milestones. (171/601; 45%), or perform a task Conclusions
more consistently (123/601; 20%). Initial ACGME milestones operationalize
Method Two approaches focused on context, the development of competence in
The authors reviewed all 25 “Milestone inferring competence from performing many ways. These findings offer insights
Project” documents available on the a task in increasingly difficult situations into how physicians understand and
ACGME Web site on September 11, (236/601; 29%) or an expanding scope assess the developmental progression
2013, using an iterative process to of engagement (169/601; 28%). Two of competence and an opportunity to
identify approaches to operationalizing used socially defined indicators of consider how different approaches may
the development of competence in the competence such as progression from affect the validity of milestone-based
milestones associated with each of 601 “learning” to “teaching,” “leading,” or assessments. The results of this analysis
subcompetencies. “role modeling” (271/601; 45%). One can inform the work of educators
approach focused on the supervisor’s developing or revising milestones,
Results role, inferring competence from a interpreting milestone data, or creating
Fifteen approaches were identified. decreasing need for supervision or assessment tools to inform milestone-
Ten focused on attributes and activities assistance (151/601; 25%). Multiple based performance measures.
T he shift in medical education disciplines.4–6 The resulting milestones on the ACGME Next Accreditation
from content- to competency-based provide insights into how competence System Web site10 on September 11, 2013;
curricula has fueled efforts to assess and its development are conceptualized these were allergy, aerospace medicine,
learners based on how well they perform in graduate medical education. colon and rectal surgery, dermatology,
various aspects of a physician’s expected diagnostic radiology, emergency
role in society.1–3 The developmental Competence is a complex construct that medicine, family medicine, general
progression of competence can be can be viewed through many different surgery, internal medicine, medical
described using milestones, the lenses.1 Likewise, the development of genetics, neurological surgery, neurology,
framework selected by the Accreditation competence can be operationalized nuclear medicine, occupational
Council for Graduate Medical Education in different ways.5,7,8 The approach medicine, ophthalmology, orthopedic
(ACGME) for reporting educational selected for a particular set of milestones surgery, pathology, pediatrics, physical
outcomes.4 To prepare for the ACGME influences their validity and, hence, may medicine and rehabilitation, plastic
Next Accreditation System, specialty have far-reaching downstream effects surgery, psychiatry, public health and
groups developed milestones for on the instruction and assessment of general preventative medicine, radiation
subcompetencies relevant to their learners, evaluation of training programs, oncology, transitional year, and urology.
and, ultimately, the confidence educators
and society can place in milestone-based The Milestone Project documents
Please see the end of this article for information
about the authors. measures of competence.9 In undertaking of these 25 specialties contained 601
this study, we aimed to identify subcompetencies (median 23, range
Correspondence should be addressed to Andrea
N. Leep Hunderfund, Mayo Clinic, Neurology, 200 approaches to operationalizing the 10–41). Of these, 37% pertained to
First Street SW, Rochester, MN 55905; telephone: development of competence in ACGME patient care (PC, 221/601), 19% to
(507) 284-4006; fax: (507) 538-6012; e-mail: leep. medical knowledge (MK, 112/601), 12%
andrea@mayo.edu.
milestones and compare approaches used
for different core competencies. to systems-based practice (SBP, 72/601),
Acad Med. XXXX;XX:00–00. 12% to professionalism (71/601), 11% to
First published online
Method practice-based learning and improvement
doi: 10.1097/ACM.0000000000001660
Copyright © 2017 by the Association of American
(PBLI, 64/601), and 10% to interpersonal
Data source and communication skills (ICS, 61/601).
Medical Colleges
Supplemental digital content for this article is We downloaded the 25 specialty-specific The origin and organization of these
available at http://links.lww.com/ACADMED/A437. “Milestone Project” documents available subcompetencies (and associated
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Research Report
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Research Report
Table 1
Approaches to Operationalizing the Development of Competence in the Initial
ACGME Milestones of 25 Specialtiesa by Core Competency
The third category of approaches used One approach in the final category focused Multiple approaches were often contained
socially defined indicators of competence, on the role of the supervisor, inferring within the milestones associated with
including progression from “learning” to competence from a decreasing need for a single subcompetency (mean 3.9, SD
“teaching,” “leading,” “role modeling,” or supervision or assistance (151/601; 25%). 1.6). Chart 2 provides an example from
“being a consultant for others” (271/601; This approach invokes the concept of the neurology milestones illustrating
45%). For example, the internal medicine scaffolding21 and was particularly explicit how three different approaches to
milestones described aspirational in the medical genetics milestones, operationalizing the development of
performance in ICS as “role models and which used the words “with substantial competence were combined within one
teaches collaborative communication guidance,” “with minimal guidance,” and set of milestones.23
with the team to enhance patient care” “independently” to describe increasing
and in PBLI as “is able to lead a quality levels of performance within many Approaches to operationalizing the
improvement project.”17 subcompetencies.22 development of competence varied by
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Research Report
Chart 1
Ability to Perform a Task Better, Faster, or in Increasingly Difficult Situations: Pediatrics Example for PC1, Gather Essential
and Accurate Information About the Patienta
Level 1 Level 2 Level 3 Level 4 Level 5
Either gathers too little Clinical experience Demonstrates an advanced Creates well-developed Creates robust illness
information or exhaustively allows linkage of signs development of pattern illness scripts that allow scripts and instance scripts
gathers information and symptoms of a recognition that leads essential and accurate (where the specific features
following a template current patient to those to the creation of illness information to be gathered of individual patients are
regardless of the patient’s encountered in previous scripts, which allow and precise diagnoses to remembered and used in
chief complaint, with patients. Still relies information to be gathered be reached with ease and future clinical reasoning)
each piece of information primarily on analytic while simultaneously efficiency when presented that lead to unconscious
gathered seeming as reasoning through basic filtered, prioritized, and with most pediatric gathering of essential and
important as the next. pathophysiology to gather synthesized into specific problems, but still relies on accurate information in
Recalls clinical information information, but has the diagnostic considerations. analytic reasoning through a targeted and efficient
in the order elicited, with ability to link current Data gathering is driven basic pathophysiology to manner when presented
the ability to gather, filter, findings to prior clinical by real-time development gather information when with all but the most
prioritize, and connect encounters allowing of a differential diagnosis presented with complex or complex or rare clinical
pieces of information information to be filtered, early in the information- uncommon problems.b,c problems. These illness
being limited by and prioritized, and synthesized gathering process.b and instance scripts are
dependent upon analytic into pertinent positives and robust enough to enable
reasoning through basic negatives, as well as broad discrimination among
pathophysiology alone.b diagnostic categories.b diagnoses with subtle
distinguishing features.b,c
Abbreviation: PC indicates patient care.
a
Reprinted with permission from The Pediatrics Milestone Project. https://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/PediatricsMilestones.pdf. Published January
2013.18 Milestone is subject to a copyright belonging to the American Board of Pediatrics, Inc. and the Accreditation Council for Graduate Medical Education.
b
Performing a given task better and/or faster (approach 2).
c
Performing a task in increasingly difficult or infrequently encountered situations (approach 11).
core competency (Table 1). For example, patients to groups of patients to and efficiently (29/61; 48%) or to
milestones associated with the 221 PC populations or systems of care; 43/72; communicate effectively in increasingly
subcompetencies often operationalized 60%) or increasing interdependence difficult situations (35/61; 57%).
the development of competence as the with other members of the health care For example, the orthopedic surgery
ability to perform different, increasingly team (e.g., progression from “dismissing” milestones describe increasing
difficult tasks (150/221; 68%) or to to “receiving” to “valuing and actively levels of performance within ICS
perform a task in increasingly difficult soliciting” input from interprofessional as communicating with patients in
or infrequently encountered situations team members18; 23/72; 32%). SBP “routine,” “difficult,” and “complex”
(137/221; 62%). PC milestones were also milestones were also the most likely to or “adversarial” situations.28 ICS and
the most likely to infer competence from use progression from “awareness” to professionalism milestones were also the
a decreasing need for supervision or “knowledge” to “ability” (36/72; 50%), most likely to infer competence from the
assistance (94/221; 43%). as illustrated by a set of pathology ability to serve as a teacher, leader, role
milestones describing the development of model, or consultant for others (54/71
Milestones associated with the 112 competence in “technology assessment” [76%] and 41/61 [67%], respectively).
MK subcompetencies usually inferred (Chart 3).27
competence from increasing knowledge
Discussion
(74/112; 84%) as evidenced by the type of Milestones related to the 71 professionalism
information known (e.g., its complexity, and 64 PBLI subcompetencies commonly In this study, we identified 15 approaches
rarity, novelty), amount of information included an attitudinal dimension. For to operationalizing the development of
known, or scores on a written test (usually example, 59% (42/71) of professionalism competence in initial ACGME milestones.
a resident in-training examination). Test milestones operationalized the development Most focused on attributes or activities
scores were defined in different ways, of competence as progression from extrinsic of learners; fewer considered social or
including a minimum percent correct,24 to intrinsic motivation or an increasing drive contextual factors or contributions from
a minimum overall raw score,20 a scaled to improve, as did 45% (29/64) of milestones supervisors. Approaches varied by core
score predictive of passing a certification related to PBLI. Professionalism milestones competency, and multiple approaches
examination,25 a passing score,26 were also the most likely to use progression were often combined within a single set
improvement in the percent correct from “unwilling” to “willing but unable” to of milestones.
compared with a previous score,14 or an “able” to perform a task (22/71; 31%) and
“acceptable” percentile ranking.14 increasing consistency with which a behavior These findings offer insights into how
or skill is demonstrated (37/71; 52%). physicians understand and assess the
Milestones associated with the 72 developmental progression of competence
SBP subcompetencies frequently Milestones associated with the 61 and provide an opportunity to consider
operationalized the development of ICS subcompetencies often inferred how different approaches may impact the
competence as an expanding scope competence from the ability to validity of milestone-based assessments.
of engagement (e.g., from individual communicate more effectively This, in turn, can inform the work of
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Research Report
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Research Report
The initial ACGME milestones have been subcompetency.35,39 However, combining processes rather than educational
referred to as “version 1.0,”9 anticipating multiple approaches without thoughtful outcomes.11 Furthermore, specialty
the need for iterative improvements and consideration risks confusing rather boards sometimes require different
revisions. Our findings provide milestones than clarifying the picture. Combining numbers to achieve competence for a
workgroups, whose initial efforts were different approaches within a single given procedure, which undercuts the
somewhat siloed within specialties,6 set of milestones also complicates validity of this approach.
with an opportunity to reexamine their the assessment process for teaching
milestones in light of approaches used by faculty (who often already struggle to Limitations of this study include the
other specialties. In doing so, educators translate milestone language into directly performance of analyses by a select
should consider which approaches to observable behaviors) and clinical group of individuals from a single
operationalizing the development of competency committees.37,40 Training institution. Additionally, approaches
competence are most likely to support the programs may similarly find it challenging to operationalizing the development
reliability, validity, and intended purposes of to implement evaluation systems of competence were coded at the
milestone-based assessments9 and use this capable of adequately assessing multiple subcompetency level rather than the
information to guide subsequent revisions. dimensions of a trainee’s performance individual milestone level. Further
(the types of tasks performed, the study is needed to determine whether
The optimal approach for a given situations in which they were performed, approaches are combined in predictable
subcompetency will likely vary depending the amount of supervision required, ways across milestone levels or whether
on the nature of the subcompetency. For the consistency of performance, etc.) certain approaches tend to co-occur.
example, progression from extrinsic to across multiple subcompetencies.6 This
intrinsic motivation may be particularly consideration is particularly important In conclusion, initial ACGME milestones
well suited for operationalizing the given the complex logistical challenges operationalize the development of
development of competence within associated with procuring longitudinal competence in many ways. This variety,
professionalism or PBLI, whereas assessment data and providing faculty both within and across specialties,
increasing interdependence with other with multiple opportunities to assess provides an opportunity to investigate
members of the health care team is well trainee performance. the relative utility, acceptability, and
suited for ICS or SBP subcompetencies meaningfulness of different approaches.9
related to teamwork. Some approaches may more readily Milestone validity may be enhanced by
lend themselves to workplace-based using approaches that are grounded in
Another important consideration is assessment than others. For example, conceptual frameworks, align with the
the level of performance expected of many milestones inferred competence cognitive schemas of raters, correlate with
incoming learners. For example, many SBP from a learner’s ability to perform a task entrustment decisions, use criterion-
milestones operationalize the development in increasingly difficult or infrequently referenced outcomes, and are informed
of competence as progression from encountered situations. By their nature, by the nature of the subcompetency and
“awareness” to “knowledge” to “ability.” such experiences may be rare and variable skill level of incoming learners.
This approach assumes that most across learners, making assessment
incoming residents will have little to strategies difficult to implement.41 In Funding/Support: This study was prepared with
no knowledge or skills related to SBP. such cases, training programs may financial support from the American Medical
Association as part of the Accelerating Change in
However, the skill level of incoming instead turn to simulation; however, this Medical Education initiative, and all participating
residents in SBP will likely increase over can be resource intensive42 and may not schools received grants through this initiative (see
time as medical schools incorporate predict performance in actual workplace www.changemeded.org for further details) and
education on health care systems and settings.43 from the Mayo Clinic Robert D. and Patricia E. Kern
systems thinking into their curricula.38 Center for the Science of Health Care Delivery. The
As the knowledge and skills of incoming Finally, we observed that some content reflects the views of the authors.
residents improve, approaches based on milestones continue to rely on
Other disclosures: None reported.
“performing a given task better or faster” approaches to assessment that have
or “performing a task in increasingly historically been more common, such Ethical approval: Reported as not applicable.
difficult or infrequently encountered as normative comparisons, increasing
experience or completion of special Previous presentations: A preliminary version
situations” may become more appropriate.
of these data was presented at the University
This highlights the dynamic nature of the training, and scores on a written test.
of Illinois at Chicago 16th Annual Masters
milestones, which are expected to evolve While approaches like these may be of Health Professions Education Summer
and change over time.9,12 appealing because of their familiarity, Conference on July 30, 2015; at the Academy
they are not well suited for a milestones for Professionalism in Health Care 4th Annual
Our findings demonstrate that multiple framework. Normative comparisons, for Conference on April 28, 2016; and as a poster at
approaches to operationalizing the example, compare trainee performance the Society of General Internal Medicine Annual
development of competence are against that of other trainees rather Meeting on May 11, 2016.
often combined within a single set than measuring performance against a
A.N. Leep Hunderfund is assistant professor of
of milestones. This may add value, as competency standard. Approaches based neurology, Mayo Clinic, Rochester, Minnesota.
different conceptual frameworks can on increasing experience (i.e., number of
D.A. Reed is associate professor of medical
offer complementary perspectives procedures performed) or completion of
education and medicine and senior associate dean of
on the developmental progression special training are likewise suboptimal academic affairs, Mayo Medical School, Mayo Clinic,
of performance within a particular because they measure educational Rochester, Minnesota.
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Research Report
S.R. Starr is assistant professor of pediatric and 13 Braun V, Clarke V. Using thematic analysis in 27 The Pathology Milestone Project. http://
adolescent medicine and director of science of health psychology. Qual Res Psychol. 2006;3:77–101. www.acgme.org/acgmeweb/portals/0/
care delivery education, Mayo Medical School, Mayo 14 The Emergency Medicine Milestone pdfs/milestones/pathologymilestones.pdf.
Clinic, Rochester, Minnesota. Project. https://www.acgme.org/ Published April 2013. Accessed September 11,
acgmeweb/Portals/0/PDFs/Milestones/ 2013, and January 18, 2017.
R.D. Havyer is assistant professor of medicine, EmergencyMedicineMilestones.pdf. 28 The Orthopaedic Surgery Milestone
Mayo Clinic, Rochester, Minnesota. Published December 2012. Accessed Project. http://www.acgme.org/
T.R. Lang is assistant professor of pediatric and September 11, 2013, and January 18, 2017. acgmeweb/portals/0/pdfs/milestones/
adolescent medicine, Mayo Clinic, Rochester, 15 The Colon and Rectal Surgery Milestone orthopaedicsurgerymilestones.pdf. Published
Minnesota (now at Gundersen Health System, Project. https://www.acgme.org/ December 2012. Accessed September 11,
LaCrosse, Wisconsin). acgmeweb/Portals/0/PDFs/Milestones/ 2013, and January 18, 2017.
ColonandRectalSurgeryMilestones.pdf. 29 Dreyfus HL, Dreyfus SE. Five Steps From
S.M. Norby is associate professor of medicine, Published April 2013. Accessed September 11, Novice to Expert: Mind Over Machine. New
Mayo Clinic, Rochester, Minnesota. 2013, and January 18, 2017. York, NY: Free Press; 1988.
16 The Allergy and Immunology Milestone 30 Anderson LW, Krathwohl DR, eds. A
References Project. https://www.acgme.org/ Taxonomy for Learning, Teaching, and
acgmeweb/portals/0/pdfs/milestones/ Assessing: A Revision of Bloom’s Taxonomy
1 Hodges BD, Lingard L, eds. The Question of allergyandimmunologymilestones.pdf. of Educational Objectives. Complete ed. New
Competence. Ithaca, NY: Cornell University Published August 2013. Accessed September York, NY: Longman; 2001.
Press; 2012. 11, 2013, and January 18, 2017. 31 Miller GE. The assessment of clinical skills/
2 Englander R, Cameron T, Ballard AJ, Dodge J, 17 The Internal Medicine Milestone Project. competence/performance. Acad Med.
Bull J, Aschenbrener CA. Toward a common http://www.acgme.org/acgmeweb/portals/0/ 1990;65(9 suppl):S63–S67.
taxonomy of competency domains for the pdfs/milestones/internalmedicinemilestones. 32 Ericsson KA, Charness N, Feltovich PJ,
health professions and competencies for pdf. Published December 2012. Accessed Hoffman RR, eds. The Cambridge Handbook
physicians. Acad Med. 2013;88:1088–1094. September 11, 2013, and January 18, 2017. of Expertise and Expert Performance. New
3 Grant G. On Competence: A Critical Analysis 18 The Pediatrics Milestone Project. https:// York, NY: Cambridge University Press; 2006.
of Competence-Based Reform in Higher www.acgme.org/acgmeweb/Portals/0/ 33 Hatano G, Inagaki K. Two courses of
Education. San Francisco, CA: Jossey-Bass; PDFs/Milestones/PediatricsMilestones.pdf. expertise. In: Stevenson HAH, Hakuta K, eds.
1979. Published January 2013. Accessed September Child Development and Education in Japan.
4 Nasca TJ, Philibert I, Brigham T, Flynn 11, 2013, and January 18, 2017. New York, NY: Freeman; 1986:262–272.
TC. The next GME accreditation system— 19 Kruger J, Dunning D. Unskilled and unaware 34 Vygotsky L. Mind in Society: The Development
Rationale and benefits. N Engl J Med. of it: How difficulties in recognizing of Higher Psychological Processes. Cambridge,
2012;366:1051–1056. one’s own incompetence lead to inflated MA: Harvard University Press; 1978.
5 Hicks PJ, Schumacher DJ, Benson BJ, et self-assessments. J Pers Soc Psychol. 35 Bordage G. Conceptual frameworks
al. The pediatrics milestones: Conceptual 1999;77:1121–1134. to illuminate and magnify. Med Educ.
framework, guiding principles, and 20 The Ophthalmology Milestone Project. 2009;43:312–319.
approach to development. J Grad Med Educ. https://www.acgme.org/acgmeweb/Portals/0/ 36 Crossley J, Jolly B. Making sense of work-
2010;2:410–418. PDFs/Milestones/OphthalmologyMilestones. based assessment: Ask the right questions, in
6 Swing SR, Beeson MS, Carraccio C, et al. pdf. Published May 2013. Accessed the right way, about the right things, of the
Educational milestone development in the first September 11, 2013, and January 18, 2017. right people. Med Educ. 2012;46:28–37.
7 specialties to enter the Next Accreditation 21 Sanders D, Welk DS. Strategies to scaffold 37 ten Cate O, Scheele F. Competency-based
System. J Grad Med Educ. 2013;5:98–106. student learning: Applying Vygotsky’s zone postgraduate training: Can we bridge the gap
7 Green ML, Aagaard EM, Caverzagie KJ, of proximal development. Nurse Educ. between theory and clinical practice? Acad
et al. Charting the road to competence: 2005;30:203–207. Med. 2007;82:542–547.
Developmental milestones for internal 22 The Medical Genetics and Genomics 38 Gonzalo JD, Dekhtyar M, Starr SR, et al.
medicine residency training. J Grad Med Milestone Project. http://www.acgme. Health systems science curricula in
Educ. 2009;1:5–20. org/acgmeweb/portals/0/pdfs/milestones/ undergraduate medical education: Identifying
8 Englander R, Hicks P, Benson B; Pediatric medicalgeneticsmilestones.pdf. Published and defining a potential curricular framework.
Milestone Project Working Group. Pediatrics January 2013. Accessed September 11, 2013, Acad Med. 2017;92:123–131.
milestones: A developmental approach to the and January 18, 2017. 39 Harris I. Deliberative inquiry: The arts
competencies. J Pediatr. 2010;157:521–522, 23 The Neurology Milestone Project. https:// of planning. In: Short EC, ed. Forms of
522.e1. www.acgme.org/acgmeweb/portals/0/ Curriculum Inquiry. Albany, NY: State
9 Holmboe ES, Yamazaki K, Edgar L, et al. pdfs/milestones/neurologymilestones.pdf. University of New York; 1991:285–307.
Reflections on the first 2 years of milestone Published June 2013. Accessed September 11, 40 Beeson MS, Warrington S, Bradford-
implementation. J Grad Med Educ. 2013, and January 18, 2017. Saffles A, Hart D. Entrustable professional
2015;7:506–511. 24 The Urology Milestone Project. https:// activities: Making sense of the emergency
10 Accreditation Council for Graduate www.acgme.org/acgmeweb/Portals/0/PDFs/ medicine milestones. J Emerg Med.
Medical Education. Milestones. https:// Milestones/UrologyMilestones.pdf. Published 2014;47:441–452.
www.acgme.org/acgmeweb/tabid/430/ December 2012. Accessed September 11, 41 Leep Hunderfund AN, Rubin DI, Laughlin
ProgramandInstitutionalAccreditation/ 2013, and January 18, 2017. RS, et al. Validity and feasibility of the
NextAccreditationSystem/Milestones.aspx. 25 The Family Medicine Milestone Project. EMG direct observation tool (EMG-DOT).
Accessed September 11, 2013, and January 18, https://www.acgme.org/acgmeweb/Portals/0/ Neurology. 2016;86:1627–1634.
2017. PDFs/Milestones/FamilyMedicineMilestones. 42 Qayumi K, Pachev G, Zheng B, et al. Status
11 Swing SR. The ACGME Outcome Project: pdf. Published April 2013. Accessed of simulation in health care education: An
Retrospective and prospective. Med Teach. September 11, 2013, and January 18, 2017. international survey. Adv Med Educ Pract.
2007;29:648–654. 26 The Transitional Year Milestone Project. 2014;5:457–467.
12 Philibert I, Brigham T, Edgar L, Swing S. https://www.acgme.org/acgmeweb/portals/0/ 43 Baig LA, Violato C, Crutcher RA. Assessing
Organization of the educational milestones pdfs/milestones/transitionalyearmilestones. clinical communication skills in physicians:
for use in the assessment of educational pdf. Published May 2013. Accessed Are the skills context specific or generalizable.
outcomes. J Grad Med Educ. 2014;6:177–182. September 11, 2013, and January 18, 2017. BMC Med Educ. 2009;9:22.
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.