Assessing the Quality

of Distance Education Programs:

A Few Items for Your Consideration

Clayton R. Wright, PhD

crwr77@gmail.com

Clayton R. Wright 1
 What is quality?

I need your help to define it.

What is the best cookie?

How do you know that it is the best?

Clayton R. Wright 2
What cookie would you choose?

Quality means different things to different

people and usually involves a comparison.

Suppose the cookie was a

treat for a deserving child?

Clayton R. Wright 3
Suppose the cookie was for a child
in a refugee camp or a rural school?
Clayton R. Wright 4
 Defining quality

• The standard of something when

compared to other similar items.

• The personnel perception of excellence.

Clayton R. Wright 5
 Quality could be defined as:

“Fitness for purpose

at a minimum cost to society.”

John Daniels: Quality Assurance and Cross Border Higher Education in the
Commonwealth: The Work of the Commonwealth of Learning at the Regulation of Cross-
Border Higher Education: Issues and Trends Workshop co-organized by UNESCO and the
Ministry of HRD in New Delhi, India, September 21-22, 2006
Clayton R. Wright 6
Why do we measure quality?
• Learner effectiveness or achievement
• Student satisfaction
• Faculty satisfaction
• Accessibility
• Student retention and graduation rates
• Accreditation and credit transfers
• Student employability or pursuit of further education
• Cost effectiveness
• Accountability to primary stakeholders
• Reputation and credibility (Dhaka)

Clayton R. Wright 7
By measuring quality,
we obtain an indication
of our success, but
we also determine
what we need to improve.

Clayton R. Wright 8
 What should be measured?

Quality may have institutional, local, national,

and international perspectives or context of use.

Clayton R. Wright 9
 Whatever
is measured
should touch on
every aspect of the
student experience.

Refer to the e-Learning Guild’s Handbook of e-Learning Strategy,

http://www.elearningguild.com/content.cfm?selection=doc.817
Clayton R. Wright 10
Approaches to measuring quality

• The mystical view

• The reputation view
• The resources/input view
• The outcomes view
• The value-added view

Valerie Lopes, Professor, Academic e-Learning, Seneca College

Clayton R. Wright 11
 A systematic approach
to examining quality

• Inputs – resources required to support program

• Processes – how programs are offered

• Outputs – results of the inputs and processes

• Outcomes – long-range, qualitative impacts

The above is filtered thru the lens of “fitness

for purpose at a minimum cost to society”.
Clayton R. Wright 12
If we focus on outcomes, we can ask:
Did we achieve the goals set by our
institution? Can we achieve the same
benefits, but for lower costs?
Did we have an impact on our community
and our society for the better?
Are we measuring the right goals?

Clayton R. Wright 13
 21st Century Learning Outcome Goals
(validated by 260 colleges)

• Technology Skills – computer literacy, internet skills

• Communication – reading, writing, speaking, and listening
• Computational – reasoning and applying mathematical concepts
• Critical Thinking and Problem-solving – decision making
• Information Management – analyzing and organizing info
• Interpersonal – building relationships, developing teams
• Personal – managing change, learning to learn, taking responsibility
• Community – ethical skills, citizenship, environmental awareness

League for Innovation and the Pew Charitable Trusts. (2000) Learning Outcomes for
the 21st Century. Mission Viejo, California.
http://www.league.org/league/projects/pew/
Clayton R. Wright 14
 Harvard University: program goals

• Equip students for participation in civic life

• Teach students to understand themselves in
terms of traditions of art, ideas, and values
• Prepare students to grapple with change
• Develop students'
understanding of the
ethical dimensions of
what they say and do

Ehrlich, T. (2007). Carnegie

Foundation Perspectives, Posting #36.
Stanford, CA. Carnegie Foundation for
the Advancement of Teaching.
Clayton R. Wright 15
 Knowing that:
• By the age of 38, today’s learner will have 10 -14 jobs.
• The amount of technical information doubles every 2 years.
• We are currently preparing students for jobs that don’t exist
that will use technologies that haven’t been invented in order
to solve problems we don’t even know are problems yet.

Isn’t it worth considering measuring non-content related

goals or the goals Harvard has for its programs?

From: Fisch, K. (2007) Shift Happens,

http://youtube.com:80/watch?v=pMcfrLYDm2U or shifthappens.wikispaces.com

Clayton R. Wright 16
 Measuring quality
• Self assessment and internal peer review
• Review by an internal or external panel
• Review of various statistics
that indicate performance and compare the results to benchmarks

• Interviews and surveys

of staff, learners, instructors, graduates, employers, and accreditation
bodies

Clayton R. Wright 17
 What challenges can we expect?

• Time – To establish and implement quality assurance systems

• Expertise – Finding committed staff with expertise
• Buy-in – Recognizing the importance of quality, having ownership
• Employment of a large number of part-time faculty
• Providing engaging material
• Providing timely, positive, and nurturing feedback
• Finance – Attaining 60%-80% of face to face costs

Clayton R. Wright 18
 Suggestions for improving
quality assurance
• Establish course development, delivery,
student support, and logistics that are
congruent with your institution’s goals,
mission, values/ethics, and community
involvement.
Be cognizant of words such as
“accessibility, equity, flexibility, and
just-in-time learning”.

Clayton R. Wright 19
• Determine who your students are
and their attributes then customize
the materials, learning support, and
logistics that will meet their needs.

• Attain the same benchmarks as

traditional, face-to-face programs
and go beyond these benchmarks
-include digital media skills, etc.

Clayton R. Wright 20
• Look at the outputs

“We need to look much more at outputs if we

seek to inculcate a culture of quality, and this
inevitably means more and better record
keeping and analysis, and more and better
research and evaluation.”
From: Tony Bates, Perspectives on Distance Education:
Towards a Culture of Quality published by COL
Clayton R. Wright 21
• Establish an informal network to review
each other’s programs.

• Provide continuous opportunities for

faculty and staff to participate in
professional development opportunities.

Quality Matters, Inter-Institutional Quality Assurance

in Online Learning, http://www.qualitymatters.org

Clayton R. Wright 22
• Ensure that qualified editors review and
proofread the materials. (Colombo to Bangkok)

• Assign someone whose primary

responsibility is to monitor and research
quality.

Clayton R. Wright 23
• Plan for future
possibilities that you
hope will never
happen.

Spend more time

asking the “what
if…” questions.
(from Dhaka)

Develop decision-
tables. (UVic)

Clayton R. Wright 24
• Realize that quality is not a static end-
point, but a dynamic, evolving process that
has an output that is constantly changing.

• Commit an identifiable
amount of resources to
initiatives that focus on
learners and learning.

Clayton R. Wright 25
Effective e-learning courses:
• Are interactive and engage
the learner
• Encourage critical thinking,
problem-solving, and
creativity
• Provide opportunities for
online practice and
knowledge transfer
• Offer timely, constructive,
relevant, and frequent
feedback
• Provide links to resources
beyond the content

Clayton R. Wright 26
It is not only possible for distance
and e-learning programs to meet
existing standards for traditional
institutions, but to exceed them.
(UKOU)

The OU has an enrolment of 200,000

and was ranked 5th among all English
Universities for teaching quality.
Clayton R. Wright 27
 Quest for excellence
Understanding the concept
Action-orientation
Learner-centric approach
Innovation for change
Training to build competencies
Year-round activity

National Assessment and Accreditation Council, India: Seven Steps to Quality

Clayton R. Wright 28
 Quality in
Distance Education Programs
For a more detailed discussion, contact:
Clayton R. Wright, PhD
crwr77@gmail.com

Clayton R. Wright 29