Accepted Manuscript

Responsible Milestone-Based Educational Handover with Individualized Learning

Plan from Undergraduate to Graduate Pediatric Medical Education

Jocelyn Huang Schiller, MD, Heather L. Burrows, MD,, Amy E. Fleming, Meg G.
Keeley, M.D, Lauren Wozniak, Sally A. Santen, MD,

PII: S1876-2859(17)30490-4
DOI: 10.1016/j.acap.2017.09.010
Reference: ACAP 1096

To appear in: Academic Pediatrics

Received Date: 25 April 2017

Revised Date: 21 August 2017
Accepted Date: 2 September 2017

Please cite this article as: Schiller JH, Burrows HL, Fleming, AE, Keeley MG, Wozniak L, Santen
SA, Responsible Milestone-Based Educational Handover with Individualized Learning Plan from
Undergraduate to Graduate Pediatric Medical Education, Academic Pediatrics (2017), doi: 10.1016/
j.acap.2017.09.010.

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to
our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and all
legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT
Title: Responsible Milestone-Based Educational Handover with Individualized Learning Plan from
Undergraduate to Graduate Pediatric Medical Education

Jocelyn Schiller, MD, Heather Burrows, MD, PhD, Amy Fleming, MD, MHPE, Meg Keeley, MD, Lauren
Wozniak, Sally A Santen, MD, PhD

Corresponding Author:
Jocelyn Huang Schiller, MD
Department of Pediatrics
University of Michigan Medical School

PT
1540 E. Hospital Drive
Mott 12-525A/SPC 4280
Ann Arbor, MI 48109

RI
Phone: 734-763-5359
FAX: 734-647-5624
johuang@umich.edu

SC
Authors:
Heather L Burrows, MD, PhD Lauren Wozniak, MPH
Department of Pediatrics University of Michigan Medical School

U
University of Michigan Medical School 2635 Gladstone Ave.
3237 Medical Professional Building Ann Arbor, MI 48104
AN
Ann Arbor, MI 48109 lwoz@umich.edu
armadill@umich.edu
Sally A Santen, MD, PHD
Amy E. Fleming, MD Department of Emergency Medicine
M

Department of Pediatrics University of Michigan Medical School

Vanderbilt University School of Medicine 6121 Taubman Health Sciences Library
201 Light Hall 1135 Catherine Street
D

Nashville, TN 37232 Ann Arbor, MI 48109-5726

amy.e.fleming@Vanderbilt.edu ssanten@umich.edu
TE

Meg G. Keeley M.D.

Department of Pediatrics
University of Virginia Health System
EP

PO Box 800739
Charlottesville, VA 22908
mmg4z@virginia.edu
C

Key Words: Medical student, pediatrics, graduate medical education, handover, milestones
AC

Running title: Responsible Milestone Educational Handover for Pediatric Students

Text word count:

Funding: The authors would like to thank the University of Michigan Graduate Medical Education
Innovations Program Grant for funding to support the dissemination of our work at the Council on
Medical Student Education in Pediatrics Annual Meeting 2017 and Pediatric Academic Societies Annual
Meeting 2017. University of Michigan Medical School has an Accelerating Change in Medical Education
Grant from the American Medicine Association. The authors have no potential conflicts of interest or
corporate sponsors.
1
 ACCEPTED MANUSCRIPT
What’s New?

Communication of medical students’ pediatric milestone assessments and individual learning

plans from medical schools to pediatric residency directors allows for effective educational

handovers promoting the continuum of education. Existing undergraduate medical education

PT
assessments can provide meaningful data to determine most pediatric milestone levels.

RI
Background

SC
Milestones are competency-based, developmental outcomes marking progress from

medical school through residency into practice. The Accreditation Council for Graduate Medical

U
Education (ACGME) defined milestones for all specialties including pediatrics.1 As medical
AN
education moves towards competency-based assessment, there are calls for a continuum,

including communication between undergraduate (UME) and graduate medical education

M

(GME).2
D

Currently, the UME to GME educational handover is the residency application, including
TE

the Medical Student Performance Evaluation (MSPE), transcripts and reference letters. The

MSPE is created almost a year before internship, without an update. Also, there are concerns
EP

regarding the trustworthiness of the MSPE.2 Although transparency may improve with the 2016

MSPE recommendations, most schools do not include competencies, and if included, do not
C

provide the detail needed to determine specialty specific milestones.3 These communication
AC

failures hinder learners’ education, wastes educational resources and may cause harm to

patients.2

To address these issues, one institution’s departments of emergency medicine, surgery,

and obstetrics/gynecology developed specialty specific, milestone-based competency

2
 ACCEPTED MANUSCRIPT
handovers between UME and GME after the National Resident Matching Program Match.4-6

Such efforts in pediatrics have yet to be described.

We sought to develop a Pediatric Milestone Handover with individual learning plan

(MH-ILP) to communicate with residency program directors (PDs) after the Match. Using a

competency-based framework,7 we shared students’ attainment of pediatric ACGME

PT
milestones. Pediatric milestones to allow schools and residencies to share a framework for

RI
assessment and communication. Using self-determination theory, ILPs were added to

encourage student self-improvement through goal setting.1,8,9 In this study, we determined the

SC
feasibility and utility of the MH-ILP from the perspective of pediatric PDs.

U
AN
Educational Approach and Innovation

In 2016, graduating students entering pediatric residencies were identified at three

M

schools. Students voluntarily participated; written consent was obtained. This study was

approved by each institution’s Institutional Review Boards.

D

Pediatric competency assessment committees comprised of PDs, clerkship directors,

TE

and internship readiness course faculty were established at each school. In spring 2016,
EP

committees reviewed the performance of their school’s participating students. After

investigators discussed data sources, each school’s committee used their own understanding of
C

milestones to plot students. Data included performance assessments from pediatric courses
AC

(clerkships, acting internships, fourth year electives, and internship readiness courses) and

observed structured clinical examinations. By consensus, committees mapped students’

assessments to the milestones and determined milestone level (Online Appendix).1 Milestones

without data were marked as “not assessed.” Although 49 pediatric milestones exist, we used

3
 ACCEPTED MANUSCRIPT
the 21 milestones required for ACGME reporting due to limited committee experience with the

other milestones.

Students were trained on the development of Specific, Measurable, Attainable, Realistic,

Timely goals. Students performed self-assessments using the milestones and set learning goals.

Most met with a committee member to compare their self-assessment with committee

PT
assessment and further refine their ILP. Two investigators reviewed the ILPs and discussed the

RI
data, codes, and themes until they reached consensus.

MH-ILP included committee milestone assessments and student–identified learning

SC
goals. The post-Match MH-ILP was sent to their residency PD with an explanation and a survey.

U
PDs were asked about the ability of the MH-ILP and MSPE to describe students’ strengths and
AN
weaknesses on a Likert scale (1= not at all, 5= extremely well) (Table 1). A similar survey was

sent 6 months later.

M

Results
D

MH-ILPs were created for 27 students who matched into 19 different programs.
TE

Committees were able to consistently assess 16 of the 21 pediatric milestones. Students were
EP

assessed to be performing at level one or greater. None of the schools consistently assessed the

milestone “Practice-Based Learning and Improvement 3” regarding quality improvement.

C

Students developed ILP goals aimed at the first internship months. Although students’
AC

141 goals covered all ACGME competency domains, the three most common sub-competencies

chosen were perform appropriate learning activities (14%), identify strengths and weaknesses

(13%) and organize/prioritize care (13%).

Sixteen PDs (84%) responded to our initial survey; 10 (53%) responded to the follow-up

(Table). In the follow-up survey, PDs felt the MH-ILP described students’ strengths and
4
 ACCEPTED MANUSCRIPT
weaknesses moderately well (strengths mean 3.9, weaknesses mean 3.4). Milestones

correlated moderately with the trainee’s mid-internship milestone assessments (mean 3.9).

One respondent commented the MH-ILP “correlated with their milestones here” and another

PD noted, “As a confirmation, seems to be continuing same positive trend,” lending support for

our milestone level assessments. Most (81%) felt the new MH-ILP provided more useful

PT
information over the traditional MSPE.

RI
Discussion and Next Steps

SC
This innovation demonstrates graduating students entering pediatrics can be assessed

U
on the majority of the pediatric required milestones and handover from UME to GME is
AN
moderately useful to pediatric PDs. Importantly, the MH-ILP provided better description of the

trainee compared to the MSPE.

M

The milestone framework1,7 was helpful in interpreting our data and confirming our

observations that these graduates were competent to begin residency and student
D

performance varied. Although we did not record the time spent collecting and reviewing
TE

student data, there was significant effort involved, which is likely to be streamlined with
EP

experience. We do believe, however, that communication between UME and GME will improve

the trainees’ educational continuum. Although PDs had mixed responses to the MH-ILP, as
C

medical education continues to dialogue about competencies, our shared understanding will
AC

increase. If more schools communicated on this level, program directors would come to expect

the information. They could use the information to tailor the intern year, perhaps mitigating

some of the “July effect” and monitoring progress of learners to help them achieve learning

goals.10

5
 ACCEPTED MANUSCRIPT
We are working to improve validity evidence; for some of the competency domains

there was rich assessment data, in others there was less robust data. The small n and lower

response rate may reflect response bias and limits generalizability of our findings, so larger

studies are indicated. For the MH-ILP to be useful, broader use at more institutions and

consistent practice standards will need to be implemented.

PT
In conclusion, existing UME assessments can provide meaningful data to determine

RI
most pediatric milestone levels. The MH-ILP provides an opportunity to improve the

assessment of learners and contribute to communication between UME and GME, further

SC
developing the continuum of learning.

U
AN
M
D
TE
C EP
AC

6
 ACCEPTED MANUSCRIPT
References:

1
The Accreditation Council for Graduate Medical Education and the American Board of Pediatrics. The Pediatric

Milestone Project. 2015. Retrieved from https://acgme.org/Portals/0/PDFs/Milestones/PediatricsMilestones.pdf

2
Warm EJ, Englander R, Pereira A, Barach P. Improving learner handovers in medical education. Acad Med.

PT
2017;92(7):927-31.

3
Association of American Medical Colleges. Recommendations for revising the medical student performance

RI
evaluation.

https://www.aamc.org/download/470400/data/mspe-recommendations.pdf; 2016 accessed 25.07.17.

SC
4
Sozner CN, Lypson ML, House JH et al. Reporting achievement of medical student milestones to residency

U
program directors: An educational handover. Acad Med. 2016;91:676-684.
AN
5
Wancata LM, Morgan H, Sandhu G et al. Using the ACGME milestones as a handover tool from medical school to

surgery residency. J Surg Educ. 2017;74(3):519-29.

M

6
Morgan H, Skinner B, Marzano D, et al. Bridging the continuum: Lessons learned from creating a competency-
D

based educational handover in obstetrics and gynecology. Med Sci Educ. 2016;26:443-447.
TE

7
Frank JR, Snell LS, ten Cate O, et al. Competency-based medical education: theory to practice. Med Teach.

2010;32:638-45.
EP

8
ABIM Foundation, ACP-ASIM Foundation and European Federation of Internal Medicine. Medical professionalism

in the new millennium: A physician charter. Ann Intern Med. 2002;136:243-246.

C

9
Accreditation Council for Graduate Medical Education. ACGME program requirements for graduate medical
AC

education in pediatrics. 2016. Retrieved from

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/320_pediatrics_2016.pdf

10
Pangaro L. “Forward feeding” about students’ progress: More information will enable better policy. Acad Med.

2008;83:802-803.

7
 ACCEPTED MANUSCRIPT

Acknowledgements: The authors would like to thank the University of Michigan Graduate Medical
Educations Innovations Program Grant for funding to support the dissemination of our work at the
Council on Medical Student Education in Pediatrics Annual Meeting 2017 and Pediatric Academic
Societies Annual Meeting 2017. The authors would also like to thank the competency committee
members Kim Lomis, Mark Mendelsohn, Jeremy Middleton, David Stewart, Rebecca Swan and Linda
Waggoner-Fountain.

PT
RI
U SC
AN
M
D
TE
C EP
AC

8
 ACCEPTED MANUSCRIPT

Table 1: Pediatric residency program director perceptions of the Medical Student Performance
Evaluation (MSPE) and Milestone Handover-Individual Learning Plan (MH-ILP)

Six month follow-up survey

Initial survey average (n=16, average(n=10, 53% response
Survey Questions 84% response rate) rate)

PT
How well did the MSPE
describe new trainees'
strengths?A 4.0 (0.65) 4.2 (0.63)

RI
How well did the MSPE
describe new trainees'
weaknesses? A 2.8 (0.91) 3.1 (1.20)
How well does the MSPE

SC
describe level one milestones
of a new trainee? A 2.4 (0.89) N/A
How well did the MH-ILP

U
describe a new trainee's
strengths? A 3.5 (0.97) 3.9 (1.10)
How well did the MH-ILP
AN
describe a new trainee's
weaknesses? A 3.0 (0.93) 3.4 (0.97)
How accurately did the MH-
ILP describe milestones of
M

your new trainee(s)? A 3.6 (1.26) 3.8 (0.92)

How well the MH-ILP
correlate with the trainee's
D

mid-internship milestone
assessment? A N/A 3.9 (0.88)
TE

50% did not use information;

19% early intervention for an 10% early intervention for an
area of weakness; area of weakness;
13% advanced curriculum for 10% used to confirm existing
EP

an area of strength; 44% no opinion;

How did you use the MH-ILP change; 10% correlated with
in your approach to the new 31% uncertain; residency milestones.
trainee(s)? 19% other 20% for "information only"
C

Does the MH-ILP provide you

with more useful information
AC

over the traditional MSPE? 81% yes, 19% no N/A

The MH-ILP provided useful
information in addition to the
traditional MSPE B N/A 3.7 (1.42)
Data are presented as mean (SD) or % except where otherwise noted.

A- Likert scale, 1= not at all, 2= slightly, 3=somewhat, 4=moderately, 5=extremely well

B- Likert scale, 1= strongly disagree, 2= disagree, 3= neutral, 4=agree, 5= strongly agree

N/A= not applicable