Original Article
Teaching Evidence-Based Psychiatry:
Integrating and Aligning the Formal and Hidden Curricula
Sacha Agrawal, M.D., Peter Szatmari, M.D., M.Sc.
Mark Hanson, M.D., M.Ed.
Objective: The authors argue that adopting evidence-based psy-
chiatry will require a paradigm shift in the training of psychiatry
residents, and offer some suggestions for how this transformation
might be achieved.
Methods: The authors review the growing literature that ad-
dresses how best to teach evidence-based medicine and highlight
several examples of innovative instructional and assessment
methods.
Results: Little is known about how best to instill among resi-
dents the attitudes, knowledge, skills, and behaviors that are nec-
essary to practice evidence-based psychiatry. However, there are
indications that the integration of evidence-based medicine in-
struction into routine clinical care and the alignment of the “hid-
den curriculum” with evidence-based practice are important.
Conclusion: A whole-program approach may be necessary to
create the conditions required in postgraduate training to pro-
duce evidence-based psychiatrists.
Academic Psychiatry 2008; 32:470–474
Received August 10, 2007; revised October 28, 2007, and January 8,
2008; accepted February 13, 2008. The authors are affiliated with
Psychiatry and Behavioural Neurosciences at McMaster University
in Ontario, Canada. Address correspondence to Sacha Agrawal,
M.D., McMaster University, Psychiatry and Behavioural Neurosci-
ences, 2757 King Street East, Hamilton, Ontario L8G 5E4, Canada;
agrawas@mcmaster.ca (e-mail).
Copyright 䊚 2008 Academic Psychiatry
470 http://ap.psychiatryonline.org
E vidence-based medicine (EBM) “represents the in-
tegration of best research evidence with clinical ex-
pertise and the patient’s unique values and circumstances”
(1). Its adoption as an important organizing principle in
all areas of clinical medicine—while not without criticisms
(2)—has profound implications for psychiatric postgradu-
ate training. Although the recently revised requirements
for residency education in psychiatry from the Royal Col-
lege of Physicians and Surgeons of Canada and the Ac-
creditation Council for Graduate Medical Education
(ACGME) stipulate that trainees in Canada and the
United States achieve competency in evidence-based prac-
tice, the method by which this goal can be achieved and
its implications for training programs are not described.
This article attempts to address this issue by considering
the meaning of evidence-based competency in the context
of psychiatry training, reviewing the available evidence
that addresses how evidence-based competency can best
be taught, and providing some examples of innovative
EBM instruction and assessment methods. We draw a par-
allel to other aspects of professional development and in-
voke the concept of the hidden curriculum (3, 4) to help
elucidate the necessary training environment.
Evidence-Based Psychiatry: A New Paradigm for
Teaching and Learning
To achieve competency in evidence-based practice, psy-
chiatry residents must learn the content and process of
EBM. The content of evidence-based psychiatry (EBP) is
the sum of the important outcomes of clinical research that
should inform psychiatric practice. This database repre-
sents essential empirical knowledge about risk, diagnosis,
treatment, and prognosis that is needed to practice psy-
chiatry effectively. Although the quantitative nature of this
information and its origin in health research differentiate
it somewhat from the essential knowledge of other para-
digms, its acquisition during training may for the most part
involve similar learning behaviors (e.g., reading textbooks
Academic Psychiatry, 32:6, November-December 2008

and articles; attending rounds and seminars). However, be-
cause this essential knowledge is continuously being mod-
ified by research advances, residents also need to learn the
process of EBP, that is, they must master the tasks of for-
mulating questions and finding answers from the research
literature. Engendering this professional behavior of con-
tinuously updating and expanding one’s knowledge of the
empirical basis for practice represents a profound chal-
lenge to psychiatric education.
For example, a senior resident assesses a female adult
survivor of childhood sexual abuse who suffers from treat-
ment-resistant major depression. The trainee is competent
in evidence-based practice and could be expected to for-
mulate this woman’s presentation by considering the quan-
titative risk posed by the experience of childhood sexual
abuse for the later development of depression, the relative
effectiveness of the available treatments for depression,
whether treatment outcomes are modified by the experi-
ence of childhood sexual abuse, and the likelihood of full
recovery given this history. This EBP knowledge sits along-
side other important information, such as the meanings
that the patient ascribes to her early experiences and her
preferences among various treatment options. It would be
impossible for any practitioner to know the best current
evidence available to address all such issues with every pa-
tient, but the EBP trainee could be expected to efficiently
and effectively search for the answers and apply them
where necessary.
What Works in Teaching Evidence-Based
Medicine?
An important question for psychiatry educators is how
to effectively foster the attitudes, knowledge, skills, and
behaviors necessary for evidence-based practice. By
searching MEDLINE and the Cochrane Library (to July
2007) using the search terms “medical education,” “evi-
dence based medicine,” or “critical appraisal,” we identi-
fied systematic reviews of the effectiveness of interventions
designed to improve students’ ability to practice EBM.
During this search, we restricted publication type to “re-
view” and scanned reference lists of selected articles for
additional titles. English-language articles were included if
they were peer-reviewed and if they reported methods for
identifying primary studies. We identified eight reviews (5–
12) summarizing a total of 32 primary studies mostly
among residents and medical students, including one in-
volving residents in psychiatry (13). One additional review
(14) that summarized qualitative data and used a qualita-
tive methodology to analyze the results was excluded.
Academic Psychiatry, 32:6, November-December 2008
AGRAWAL ET AL.
Several observations emerge from these studies. First,
reviewers found serious methodological problems in many
or most of the primary studies examined. These method-
ological challenges, summarized by Hatala and Guyatt
(15), include difficulty establishing valid controls free of
contamination and cointerventions, small sample sizes that
were limited by the size of the medical school or postgrad-
uate program, a low signal-to-noise ratio that reflected the
small effect that educational interventions can have on
learner behavior, and a scarcity of valid and reliable out-
come measures. Particularly striking were the absence of
examples where a process or outcome of care was reliably
and validly measured as an outcome of the intervention
and the heavy reliance on self-report measures of learner
behavior.
The second observation is that educational interven-
tions appear to have had a greater influence on student
knowledge (an understanding of principles, demonstrated
usually on a multiple-choice test) than on skills (an ability
to apply knowledge by performing an EBM step) or be-
haviors (actually performing EBM in clinical practice).
These findings are consistent with what is known about
knowledge translation: imparting knowledge through
training is necessary but usually insufficient to produce be-
havioral change (16). Opportunities to practice the new
skill, coaching, and downstream evaluations of behaviors
are also typically required (17), but have generally not
been included in the educational interventions described
to date.
Third, the outcomes that can be achieved at different
training levels may differ. In one review outcomes were
stratified according to student level (10), and knowledge
of EBM was found to be consistently improved among
medical students, but was unaffected among residents.
Noting that evaluation is known to act as a major deter-
minant of learning, the authors comment that this result
may be attributable to the more direct connection between
learning and evaluation in undergraduate compared with
postgraduate medical education. Another possibility is that
residents may be less able to learn something that they may
see as irrelevant if it goes against the clinical culture in
which they operate. Put another way, interventions de-
signed to teach EBM may have little chance of success if
they do not fit well with the hidden curriculum—that is, the
set of influences that function at the level of organizational
structure and culture (4). This hidden curriculum, which
will demonstrate for students the value (or lack thereof)
of evidence-based practice by determining, for example,
the extent to which evidence-based practice is modeled by
http://ap.psychiatryonline.org 471
TEACHING EVIDENCE-BASED PSYCHIATRY
supervisors and peers, likely plays as important a role in
the acquisition of evidence-based practice as it does in
other aspects of professionalism (3).
The relationship between educational interventions and
the hidden curriculum may also help explain a fourth ob-
servation: integration of educational interventions into the
daily clinical work of students is associated with better out-
comes. In their reviews, Coomarasamy and Khan (6) found
that teaching methods that were integrated into clinical
practice (i.e., based on actual patients and clinical prob-
lems) were more often effective than those that were not
(positive results in 12/12 versus 16/34 studies, respectively).
This result is highly consistent with the broad educational
finding that context is critical for learning (6), and leading
EBM educators have emphasized the need to integrate
teaching with the provision of clinical care to maximize its
educational impact (18). It is also possible that the orga-
nizational characteristics that create hidden curricula de-
termine the likelihood that a program will adopt integrated
teaching methods. If this is true, the observed added im-
pact of integrated interventions may not be as much the
result of integration as of differences in structure and cul-
ture among programs that were present at the start. Only
a study in which whole programs are randomized to inte-
grated versus standalone interventions could clarify the re-
lationships between these variables, but for now it may be
enough for educators to acknowledge and make optimal
use of both formal and hidden curricula.
Innovative Methods for Instruction in Evidence-
Based Medicine
It has been observed that clinical questions occur fre-
quently in the minds of students but are rarely followed by
a search of the literature for an answer (19). A qualitative
study (20) of internal medicine residents yielded a number
of factors that influence whether residents find answers to
their clinical questions, including having sufficient time,
having a way of remembering and tracking questions after
leaving the clinic, having a way of prioritizing questions,
the microclimate in which the residents work, their per-
sonal initiative, access to the literature at point-of-care, the
possession of skills to find answers to questions, and the
global institutional culture. A number of innovative (albeit
mostly untested) approaches to EBM education may help
address these barriers by creating a more tightly integrated
educational program and shifting the hidden curriculum
toward a culture that fosters evidence-based psychiatry.
Richardson (21) described a model for considering how
clinical instructors teach EBM, which may represent a use-
472 http://ap.psychiatryonline.org
ful starting point for faculty development. First, teachers
model evidence-based practice themselves by being ex-
plicit when they ask questions, search for evidence, criti-
cally appraisal the literature, and when they apply evidence
to patient care. Second, teachers weave evidence into their
clinical teaching by including the results of health research
of risk, diagnosis, treatment, and prognosis in their discus-
sions with students. Third, they teach the skills needed to
practice EBM by guiding students in their own formulation
of questions and their search for and appraisal of evidence
and its application to patient care. The traditional super-
vision meeting between psychiatry resident and clinical su-
pervisor, with its built-in opportunities for reflection on
and follow-up of learning issues, is ideally suited for all
three of these modes of EBM teaching.
One method for making better use of questions as they
come up is to record them as an educational prescription
that indicates the structured clinical question, the time and
place at which the answer will be reviewed, and the expec-
tations of the learner’s presentation (22). Another way of
tracking clinical questions is to enter them into an elec-
tronic database. Two groups have published their experi-
ences asking residents to generate electronic “learning
portfolios” of such questions. Fung et al. (23) described
the implementation of a system across four Canadian ob-
stetrics and gynecology training programs that logs patient
encounters and directs reflection on critical incidents of
learning by asking users to enter clinical questions and
their answers. Crowley et al. (24) described a database,
implemented across two inpatient general medical wards,
that residents used to track clinical questions and the crit-
ically appraised answers generated while on call. Users re-
ported that the process of asking and answering questions
altered patient management in 47% of encounters. This
database was designed to allow searching by other users
and therefore served an additional function as a clinical
resource for house staff.
Although a number of evaluations of traditional journal
clubs found no effect on the skills of attendees (5), includ-
ing the only evaluation of an EBM curriculum among psy-
chiatry residents (13), it is possible that the journal club
experience may be made more effective if the process of
evidence-based practice is more faithfully rendered. So-
called “question-driven” journal clubs, in which members
start with an actual patient encounter and proceed through
the steps of asking, acquiring, appraising, applying, and
assessing, may foster better learning than when papers are
selected independent of a clinical problem at hand. Kallen
et al. (25) described a variation on the traditional journal
Academic Psychiatry, 32:6, November-December 2008

club format where fellows and consultants met twice for
every article that they considered and, in the second meet-
ing, jointly wrote a letter to the editor with their method-
ological criticisms and suggestions for improvement. The
group’s success of publishing three letters out of nine ar-
ticles is noteworthy and may have represented an impor-
tant source of motivation for participants.
Assessment of Evidence-Based Medicine Skills
A systematic review of instruments that evaluate the
teaching of EBM (26) documents the growing awareness
that a repertoire of valid and reliable instruments for mea-
suring its effects is needed to ensure that EBM education
is evidence-based. The authors of that review highlight two
instruments, the Fresno test (27) and the Berlin question-
naire (28), for their relatively well-established validity and
reliability. The web-based Fresno test starts with two clini-
cal vignettes that suggest clinical uncertainty and asks stu-
dents to demonstrate how they would formulate a clinical
question, acquire the relevant evidence, appraise that evi-
dence, and apply it. This test has the advantage of assessing
more than the critical appraisal step, which has historically
been the focus of outcome measures but may not be as
important with the growing availability of high-quality syn-
opses and other sources of preappraised evidence. The
Fresno test has been shown to have high interrater reli-
ability, internal consistency, and discriminant validity (27).
The Berlin questionnaire is a 15-item multiple-choice test
that measures EBM knowledge and skills and has been
shown to have content, discriminant, and responsive valid-
ity (28). It has the advantage of being more easily scored
than the Fresno test but asks students to demonstrate skills
in fewer EBM steps. Both instruments represent ways for
training programs to document EBM competency among
individual trainees. Alternatively, one group has published
the feasibility and results of an EBM observed structured
clinical examination station (29). The student was asked
by a standardized patient to provide evidence supporting
a clinical decision that he or she had just made. The stu-
dent was then instructed to formulate a clinical question,
perform a brief literature search in real time, appraise a
standardized article, and then present the bottom line to
the patient. This instrument is unique in providing a
method of assessing communication of evidence to the pa-
tient, but lacks the demonstration of robust psychometric
properties that would be necessary for its use as an eval-
uation method. An important next step for educators in
psychiatry is to adapt these or similar instruments to the
Academic Psychiatry, 32:6, November-December 2008
AGRAWAL ET AL.
mental health context and establish their feasibility and
psychometric properties among trainees in psychiatry.
Conclusion
The way in which health research is brought into clinical
encounters with patients will evolve as information tech-
nology continues to transform the knowledge-to-practice
pipeline. However, the relatively recent emphasis on the
use of best evidence in guiding health care will likely re-
main an important feature of psychiatry in the foreseeable
future (30). The question of how best to train residents of
any specialty to become evidence-based physicians re-
mains important and largely unanswered at this time. Still
less is known about whether and how this task may pose
unique challenges to psychiatry, although the near absence
of studies in psychiatric contexts suggests an opportunity
for more research in this area. What does seem clear from
the available data is that teaching residents to practice
EBM is not straightforward. To be incorporated into the
behavior of students, evidence-based psychiatry likely
needs to occur in the routine clinical work of residents. It
therefore needs to be adopted as an expectation of clinical
care, modeled and valued by supervisors and regularly
evaluated, in addition to being formally taught in seminars
and journal clubs. Thus achieving evidence-based compe-
tency among faculty will be a necessary condition of
achieving evidence-based competency among trainees, a
goal that naturally poses its own formidable challenges. If
producing evidence-based psychiatrists is the goal, post-
graduate programs need to examine, integrate, and align
their formal and hidden curricula to ensure a learning en-
vironment that fosters evidence-based practice.
The authors thank Karen Saperson, Allyn Walsh, Robert Zipur-
sky, and the anonymous reviewers for their helpful comments.
References
1. Sackett DL, Rosenberg WM, Gray JA, et al: Evidence-based
medicine: what it is and what it isn’t. BMJ 1996; 312:71–72
2. Straus SE, McAlister FA: Evidence-based medicine: a com-
mentary on common criticisms. CMAJ 2000; 163:837–841
3. Hafferty FW, Franks R: The hidden curriculum, ethics teach-
ing, and the structure of medical education. Acad Med 1994;
69:861–871
4. Lempp H, Seale C: The hidden curriculum in undergraduate
medical education: qualitative study of medical students’ per-
ceptions of teaching. BMJ 2004; 329:770–773
5. Alguire PC: A review of journal clubs in postgraduate medical
education. J Gen Intern Med 1998; 13:347–353
http://ap.psychiatryonline.org 473
TEACHING EVIDENCE-BASED PSYCHIATRY
6. Coomarasamy A, Khan KS: What is the evidence that post-
graduate teaching in evidence-based medicine changes any-
thing? A systematic review. BMJ 2004; 329:1017
7. Coomarasamy A, Taylor R, Khan KS: A systematic review of
postgraduate teaching in evidence-based medicine and critical
appraisal. Med Teach 2003; 25:77–81
8. Ebbert JO, Montori VM, Schultz HJ: The journal club in post-
graduate medical education: a systematic review. Med Teach
2001; 23:455–461
9. Green ML: Graduate medical education training in clinical
epidemiology, critical appraisal, and evidence-based medi-
cine: a critical review of curricula. Acad Med 1999; 74:686–
694
10. Norman GR, Shannon SI: Effectiveness of instruction in criti-
cal appraisal (evidence-based medicine) skills: a critical ap-
praisal. CMAJ 1998; 158:177–181
11. Parkes J, Hyde C, Deeks J, et al: Teaching critical appraisal
skills in health care settings. Cochrane Database Syst Rev
2001; 3:CD001270
12. Taylor R, Reeves B, Ewings P, et al: A systematic review of
the effectiveness of critical appraisal skills training for clini-
cians. Med Educ 2000; 34:120–125
13. Fu C, Hodges B, Regehr G, et al: Is a journal club effective
for teaching critical appraisal skills? A controlled trial with
residents in psychiatry. Acad Psychiatry 1999; 23:205–209
14. Bradley P, Nordheim L, De La Harpe D, et al: A systematic
review of qualitative literature on educational interventions
for evidence-based practice. Learning in Health and Social
Care 2005; 4:89–109
15. Hatala R, Guyatt G: Evaluating the teaching of evidence-
based medicine. JAMA 2002; 288:1110–1112
16. Mazmanian P, Davis D: Continuing medical education and the
physician as learner: guide to the evidence. JAMA 2002;
288:1057–1060
17. Fixsen DL, Naoom SF, Blase KA, et al: Implementation re-
search: a synthesis of the literature. Tampa, Fla, University of
South Florida, Louis de la Parte Florida Mental Health In-
stitute, National Implementation Research Network, 2005
474 http://ap.psychiatryonline.org
18. Hatala R, Keitz SA, Wilson MC, et al: Beyond journal clubs:
moving toward an integrated evidence-based medicine curric-
ulum. J Gen Intern Med 2006; 21:538–541
19. Green ML, Ciampi MA, Ellis PJ: Residents’ medical infor-
mation needs in clinic: are they being met? Am J Med 2000;
109:218–223
20. Green ML, Ruff TR: Why do residents fail to answer their
clinical questions? A qualitative study of barriers to practicing
evidence-based medicine. Acad Med 2005; 80:176–182
21. Richardson WS: Teaching evidence-based practice on foot.
ACP J Club 2005; 143:A10–12
22. Straus SE: Evidence-based medicine: how to practice and
teach EBM, 3rd ed. Edinburgh; New York, Elsevier/Churchill
Livingstone, 2005
23. Fung MF, Walker M, Fung KF, et al: An internet-based learn-
ing portfolio in resident education: the KOALA multicentre
programme. Med Educ 2000; 34:474–479
24. Crowley SD, Owens TA, Schardt CM, et al: A web-based com-
pendium of clinical questions and medical evidence to educate
internal medicine residents. Acad Med 2003; 78:270–274
25. Kallen AJ, Wilson CT, Russell MA, et al: Group writing of
letters to the editor as the goal of journal club. JAMA 2006;
296:1053–1054
26. Shaneyfelt T, Baum KD, Bell D, et al: Instruments for eval-
uating education in evidence-based practice: a systematic re-
view. JAMA 2006; 296:1116–1127
27. Ramos KD, Schafer S, Tracz SM: Validation of the Fresno
test of competence in evidence based medicine. BMJ 2003;
326:319–321
28. Fritsche L, Greenhalgh T, Falck-Ytter Y, et al: Do short
courses in evidence-based medicine improve knowledge and
skills? Validation of Berlin questionnaire and before and after
study of courses in evidence-based medicine. BMJ 2002;
325:1338–1341
29. Davidson RA, Duerson M, Romrell L, et al: Evaluating evi-
dence-based medicine skills during a performance-based ex-
amination. Acad Med 2004; 79:272–275
30. Institute of Medicine (U.S.). Committee on Crossing the
Quality Chasm: Adaptation to Mental Health and Addictive
Disorders. Improving the quality of health care for mental and
substance-use conditions. Washington, DC, National Acade-
mies Press, 2006
Academic Psychiatry, 32:6, November-December 2008