Chapter 10: Evaluation

Tomica Bozic, Hajrija Seleskovic, Soeren, Huwendiek, Martina Kadmon

1. Introduction
Evaluation is an essential part of the education process. It includes evaluation of
teaching of the individual teacher and of the program. It provides improved teaching
and its results in better learning for the students, improved care for the patient and a
better educational program.
Purpose of evaluation:
To ensure teaching is meeting students learning needs
To identify areas of improving teaching
To allocate faculty resources
To provide feedback for teachers
To support promotions by teachers
To identify the values by medical schools
To support developement of the curriculum
When an evaluation is planned we are asking?
What are the goal of the evaluation?
From whom and in what will data be collected?
Who will collect and analyse data?
What type of analysis will be used?
Who will see the results of the evaluation?
Ideal evaluation has the following characteristics:
Reliability
Validity
Acceptability to evaluator and to persons being evaluated
Inexpensiveness
Reliability: the teaching evaluation needs to be linked to curriculum developement
and needs criteria which are consistent with a teachers job. The evaluation may
incorporate: the teachers themselves, learners, peers, supervisors, other health
professionals, program directors and patients.
Validity: where possible, more then one source of information should be used and
evaluation should include objective and subjective methods and quantitative and
qualitative information.
Acceptability: is necessary to provide opportunities for teaching practice with
feedback with rewards and having appropriate confidence. Good evaluation system is
easy to administer and to be applied to all levels of teachers. Also good evaluation
systems need to produce useful results which should be linked to professional
development. These may be educational outcomes (e.g. teacher performance, students
learning), practice outcomes (e.g. change in trainee practice) and health outcomes
(e.g. effect on patient health).
Evaluation by students is limited to areas in which they are competent to judge.
They need to feel competent for an evaluation and in its developement.
Finally, students need to know that their opinions are valued. They also have to be
informed about the results of the evaluation.
Self evaluation: academic staff evaluate their own practice; self evaluation provides
to change behavior; teachers have to find it useful to use different devices during
teaching( videotapes, logbooks).
Peer evaluation: teacherss peers can provide an informed, valuable and diagnostic
evaluation.
All teachers and residents should be evaluated in all domains : knoweledge,
scholarship, clinical competence, teaching effectiveness and professional attributes.
2. Evaluation Methods
Evaluation includes subjective and objective measures and qualitative and quantitative
approaches. They are:
Interviews: the group interview or group focus can be used to explore the
opinions of several individuals simultaneously from students and residents. An
individual interview may be used for a teacher who has received low ratings
and the reasons are unknown and not clear.
Surveys: peer review of teaching provides valuable and diagnostic evaluation
of the clinical teacher. These include faculty peers and educational
consultants. This evaluation leads to faculty development.
OSTE: For evaluation and assessment of teaching skills simulation of teaching
encounters can be used. With OSTE (objective structured teaching encounter)
clinical teaching activities can be explored. The faculty can score teaching
attributes. Aim of evaluation of teaching effectiveness is improvement of
learner outcomes.
Evaluation of teaching from different perspectives
2.1. The teachers perspective
The evaluation of teaching and teachers may be done directly or indirectly affected by
their clinical, research and administrative competencies.
Knowledge of clinical discipline: Many countries prefer continuing professional
development in the relevant domain of practice. For teachers of clinical medicine an
adequate up-to-date knowledge base is an essential characteristic. Post specialization
training certification and interest in life long learning activities can be used to reflect
this knowledge base.
Values and models scholarship: This includes the development and implementation
of a personal lifelong learning, critical appraisal of sources of medical information.
Teachers knowledge, scholarship and approach to lifelong learning can be evaluated
by peer and learner review of practice and teaching.
Assessment of learners: For the feedback of teachers this is an essential element for
teaching. Assessment of a teachers ability to evaluate is done by learners. Peers and
allied health professionals may participate in the trainee evaluation.
Improving teaching skills: Almost all academic health centers provide programs for
the improvement of teaching. The clinical teachers have to be aware of a continuing
effort to improve teaching skills. This process is determined by reviewing the teaching
dossier or the evaluation by peers involved in residency programs in medical school.
Professionalism: It includes knowledge and skills, professional ethics and humanistic
qualities. Humanistic qualities include integrity, respect, compassion, professional
responsibility, courtesy, sensitivity to the patients needs and professional attitudes
towards colleagues and learners. Professional ethics include a high standard of moral
and ethical behavior. Students, postgraduate trainees, patients, nurses and peers
evaluate professional behavioral and humanistic skills of teaching physicians.
Collegiality and collaboration: Collaboration is a teachers ability to work with
patients and multidisciplinary health professionals to provide optimal patient care,
education and research. Interdisciplinary activities are essential for teaching and
research in medicine. Students, residents, peers, allied health professionals and
patients evaluate teachers collegiality and collaboration.
Competent clinical skills and role modeling of good clinical skills: Patients, allied
health professionals, learners (students and residents), peers who contribute in the care
of the patient evaluate the performance of physicians.
2.2. The learner's perspective
The clinical teachers have to provide learners with knowledge and skills for their
future professional life. From the learners perspective excellent teachers have to be
capable of assessing the level and needs of the learner. The good teachers have to
create a positive learning environment, involving all student in the learning process,
encouraging mutual respect for all learners, discussing teachers and learners
expectations, accepting what he or she is doing and why. Characteristics of a teacher
which are important for the learner include enthusiasm, clarity, knowledge of the
discipline, and demonstrating clinical skills.
Quantitative data to evaluate are obtained by using questionnaires by learners.
Confidentiality and anonymity have to be assured. Detailed qualitative data can be
obtained from interviews or focus groups. Finally learners have to perceive that their
evaluations are taken seriously.
2.3. The patient's perspective
Mostly patients lack the knowledge to assess technical skills or medical knowledge,
but they are able to judge whether they have been treated in an ethical and
compassionate manner. Patients can be involved in the evaluation of clinical teaching
in three ways.
First, they can assess aspects of clinical teaching encounter.
Second, for patients their overall clinical care is important.This evaluation may be
related in part to a demonstration of teaching skills.
Third, for patients it is important that their physicians are courteous, that they create a
relaxed atmosphere, listen closely, understand, take patient complaints seriously, and
explain what they are doing throughout the examination. Patients can evaluate clear
and competent physicians explanations, their involvement in the planning of
treatment, physicians respect for patient dignity, and outcomes of treatment from
patients perspective.
High ratings have been given by older patients, males, less educated and healthier
patients.
The methodology of patient feedback is still evolving. - High ratings have been given
by older patients, males, less educated and healthier patients -. But the data can be
helpful as a feedback to the individual physician/teachers regarding to the nature of
the all over evaluation of the clinics and may be helpful for program administrators
and faculty development planners.
2.4. The perspective of customers and
stakeholders
The traditional stakeholders are governments and health insurance companies. The
best arrangements may exist between insurance companies, local health care
organizations and individual clinicians. They all are interested in high quality health
care and the support of medical education. The government programs are interested in
teaching evaluations. Health insurance companies are interested in minimizing costs
for health care measures.
Deans of medical schools have to be interested in hospitals as an academic
environment with high quality for teaching and research. Research activity levels,
teaching evaluations and students satisfaction may be attractive for more students
and motivated and experienced staff.
Different expectations of customers and stakeholders on outcomes of clinical
teaching:
Government: High quality of health care to meet the needs of the population
Hospital directors:
High quality in clinical service
High quality of clinicians/teachers
More patients
Higher status
Higher income
Deans:
Successful research programs
High quality teaching
International acknowledgement
Quality in health care with good outcomes
Corporations: Cost efficiency
Health insurance
Companies:
Cost control
Clinicians:
Balance between clinical and teaching roles
Reward for good teaching
Students:
Quality of teaching
Support in all fields of education and research
Patients:
Confidentiality
Respect
No time delay
Quality concerning outcomes of treatment
No increase in cost
3. Collecting Information
1. Kind of information
2. Length of questionnaire
3. Approach of evaluation
Questionnaires offer a collection of information about poeples knowledge, beliefs,
behavior, expectations, respects (and vice versa), quality (i.e. education), efficiency
(i.e. cost).
Questionnaires can be used as the only measure ( as in cross sectional survey) or
within clinical trials or epidemiological studies.
Questionnaires need to be defined according to their length (dependent of kind of
information which have to be asked about) and how it will be administered.
Questionnaires have to be defined according to the approach. Evaluation may involve
objective and subjective measures and qualitative and quantitative approaches.
Subjective measurements include qualitative (focus groups and interviews) and
quantitative (surveys) measurements.
Objective measures include qualitative (analysis of tasks completed in an objective
structured clinical examination - OSCE- station).and quantitative measures (trend in
examination scores).
Examination core is a comparison for students in the previous cohort with those
participating in the teaching session.
4. Designing Evaluation
Suitability of Questionnaires
Questionnaires will be appropriate only if used within a mixed methodological study.
This is important for participants since answers must be meaningful.
Using existing instruments
For actual measurements previously validated and published questionnaires should be
used. In this manner it will be possible to compare the actual data with those from
other studies. This procedure should not exclude the own input and achievement of
the proposed evaluation.
If there is no questionnaire available a new one has to be established.
In health service research designed standard questionnaires are used so that the data
can be compared across multi center studies. This is important for validity approach.
Health status and quality of life lose their validity when used beyond this context (for
example clinical trials).
Reliability and and validity of questionnaire
Valid questionnaires measure what it claims to measure. Reasons for failing can be
questionnaires developed too short of time without competence, in different countries
or cultures with different ethical background.
Questionnaires are reliable when results are consistent from repeated samples and
different researchers over time. Differences in results may come from differences
between participants.
Standard questionnaires increase its reliability. Participants are asked same question in
an identical format and responses recorded in a uniform manner.
Manner of question presentation
Two words which are often used are frequently and regularly. Frequently means
frequency (frequency based rating scale). Regularly means a pattern. To avoid
answers without clear statements like: commonly, usually, many, some and hardly
ever, questions like strongly agree through to strongly disagree should be used.
We can use ticking a particular box or saying yes, no or maybe. Also we can insert a
free text box at the end of the questionnaire. Participants need instructions how to
complete free text items.
If open ended questions or free text comments are used qualified experts analysing
these data are necessary.
We need to realize that some respondents ( known as yes sayers) want to agree rather
than disagree. If one question is affirmative, another question must be in the negative.
What else questionnaire should include?
It is important to provide an introductory letter or information sheet for participants
and to take it away after completing the questionnaire,
How questionnaires should look like
In principle, questions should be short, around 12 words or less. But short questions
can be perceived as inconvenient and longer sentences are preferred.
Selecting a sample
Sample size of questionnaires is determined according to the expected outcome. If we
are collecting quantitative data we will use statistical advice on the minimum sample
size.
For qualitative subjective evaluation reliable information can be obtained from 100
student, we dont need more participants.
For evaluation we need more than one source and type of information.
Approval is needed before we strart
In principle a study is unethical if it is scientifically unsound, causes undue offence or
trauma, breaches confidence or wastes peoples time or money.
Questionnaires formally have to be approved by the responsible institution (ethic
committee, university, faculty, responsible supervisor etc.) before they can used. The
responsible institutes will require details of the study design and copies of the
questionnaire.
Besides these instructions it is evident:
Questionnaire studies often fail to produce high quality data.
When possible use previously validated questionnaires
Questions have to be phrased appropriately for the target audience and
information required.
Good explanations and design will improve response rates.
Evaluation involves answering questions how did I do and what did the students
learn .
Teachers can be evaluated in different ways.
If the students are using a source of feedback, the following methods are useful:
We can ask students to read their lecture notes and we have information what
students have learned and understood.
We can ask for verbal feedback from individual students
We can ask students to complete a one-minute paper
We can ask the students to complete an evaluation questionnaire
Peers can be used for a feedback to evaluate teaching style and delivery.
Ask a colleague to observe part or all a lecture and what aspects of the
lecturing are for evaluations: clarity, logical flow, effectiveness.
5. Evaluation Form Examples


6. References
Petra MB,Trisha G. Selecting, designing and developing questionnaire.BMJ
2004;328:1312-1315.
Linda S,Susan T,Steven H, Richard H,John N, Katinka P,Artur R,Richard R. A review
of the evaluation of clinical teaching: new perspectives and challenges.Medical
education 2000;34:862-870.
Jill M. ABC of learning and teaching in medicine: Evaluation.BMJ 2003 Feb
15;326(7385):385-7.Review
Cantillon P.Teaching large groups.BMJ 2003 Feb 22;326(7386):437.Review.