Evaluating Employee Training

Effectiveness: Applying
Academic Principles for
Program-Level Learning
Outcomes Assessment
John Lavrenz, Tom Kaness, & Olin
Oedekoven, Ph.D.
 Abstract
There is a clear and compelling need to assess how well various employee-training programs meet the desired
expectations and outcomes by employers striving to provide ongoing training of their employees and the
continuous improvement of such programs. Although some evaluation may occur towards the end of a training
program, uncertainty exists regarding the precise effectiveness of the training program itself unless a different
evaluation construct is implemented. Such analyses would then drive continuous improvement efforts for the
employee-training program.

A possible evaluation construct relative to training effectiveness would be to adopt methods frequently used in
higher education in order to directly measure retained knowledge and the change of knowledge that occurs
because of the training experience. Academic officials frequently use pre-tests/post-tests for course-level
assessment. Academia also uses inbound/outbound testing to assess program-level learning outcomes. In this
construct, students are assessed at the beginning of an educational program and then re-assessed just prior to
graduation. The difference in scores relate to both measuring retained knowledge as well as measuring the
change that occurred as a direct result of the educational experience.

The purpose of the presentation is to explore how an academic construct for program-level assessment could
be applicable for employee development programs within industry. We will discuss both pre-test/post-test and
inbound/outbound assessment constructs. The goals of such an assessment program would include the
improved safety of employees, as well . Over time, such analyses would provide detailed information regarding
the overall effectiveness of the training program. The proposed approach addresses an essential and
fundamental question: are the funds spent on training actually returning a value to the company?
 Agenda
Program Assessment in Higher Education
Terminology

Indirect vs. Direct Measures of Effectiveness

The Value of Direct Measurement
The Inbound/Outbound Construct
Applications & Considerations for Industry
Program Assessment in
Higher Education

Evaluating Employee Training Effectiveness: Applying Academic

Principles for Program-Level Learning Outcomes Assessment
6th International Industry Summit on Business Process
Performance in Achieving Lasting Improvements in the
Energy and Mineral Industries, May 2013
The Accountability Imperative
The political climate is clearly impacting the way in
which higher education does business in terms of
academic accountability:
 More regulations from the US Department of Education, CHEA,

and Regional Accreditation Organizations

 State Boards of Education and State Legislatures tightening the

funding belt and looking at greater accountability in higher

education
 Demand from students, parents, and other academic

stakeholders for more proof that their higher education

experience is worth the financial costs
Business Program Accreditation
Requirements
 How the Direct Assessment
Process Works
End of Program
Start of Program Student Completes
Academic Program
Student takes the
Student takes the
OUTBOUND
INBOUND Exam,
Exam, Final
Initial Benchmark.
Benchmark. This
This is usually
is usually taken
taken during their
during the last
first program
course in the
course. The change in scores
program.
is a direct measure of
competency change,
both for the individual
and for the academic
program.
 Terminology

Evaluating Employee Training Effectiveness: Applying Academic

Principles for Program-Level Learning Outcomes Assessment
6th International Industry Summit on Business Process
Performance in Achieving Lasting Improvements in the
Energy and Mineral Industries, May 2013
Academic Terminology 1 of 3
Inbound Exam – a program-level
assessment given at the start of a
student’s program of study.
Pre-test – a test given at the start of a course
Outbound Exam – a program-level
assessment given at the end of a
student’s program of study.
Post-test – a test given at the end of a course
Academic Terminology 2 of 3
From an industry perspective, think of
program-level (Inbound/Outbound) as a
position-level series of courses meant to build
competency for an employee’s position in the
organization.

From an industry perspective, think of course-

level as a task-level test associated with
specific task competency.
Academic Terminology 3 of 3
The target audience of Inbound/Outbound
testing is the Training Department, Provider,
or Program; NOT the participant. It is
essential in this construct that you shift the
mindset of testing from one that evaluates
only from a participant-perspective, to one of
the broader, institution/program.

In other words, the target audience of

Inbound/Outbound testing is the Company,
not the participant.
 Indirect vs. Direct
Measures of Effectiveness

Evaluating Employee Training Effectiveness: Applying Academic

Principles for Program-Level Learning Outcomes Assessment
6th International Industry Summit on Business Process
Performance in Achieving Lasting Improvements in the
Energy and Mineral Industries, May 2013
Indirect Measures: More commonly used to
measure training/educational effectiveness

In Academia: In Industry
 Student  Turnover rates
Satisfaction
Surveys  Course Pass rates

 Job Placement  Employee Surveys

after graduation
 Production Results
 Grades/GPA
 Safety TRIR
 Subjective
Rankings
Direct Measures: Becoming required in
academia

In Academia: In Industry
 Pre/Post Testing  ???
 Inbound/Outbound
Testing
Training ROI

Confusing to say the least!!

 The Value of Direct
Measurement

Evaluating Employee Training Effectiveness: Applying Academic

Principles for Program-Level Learning Outcomes Assessment
6th International Industry Summit on Business Process
Performance in Achieving Lasting Improvements in the
Energy and Mineral Industries, May 2013
The Value of Direct Measurement
of Educational Effectiveness

Variable Control – with a direct

measurement construct to measure
training and education effectiveness,
fewer (usually external) variables are
at play that could bias other (usually
indirect) measures.
The Inbound/Outbound
Evaluation Construct

Evaluating Employee Training Effectiveness: Applying Academic

Principles for Program-Level Learning Outcomes Assessment
6th International Industry Summit on Business Process
Performance in Achieving Lasting Improvements in the
Energy and Mineral Industries, May 2013
Academic Research:
The Partnership Study:
Inbound/Outbound Comparisons:
Learning Outcomes Analyses:
Student-level Analyses
Longitudinal Analyses
 Applications &
Considerations for Industry

Evaluating Employee Training Effectiveness: Applying Academic

Principles for Program-Level Learning Outcomes Assessment
6th International Industry Summit on Business Process
Performance in Achieving Lasting Improvements in the
Energy and Mineral Industries, May 2013
Applications of
Inbound/Outbound Evaluation
Quantify the change of knowledge that
occurred as a direct result of the
training/education experience.
Identify both the strengths and
opportunities for improvement of the
training program and for individual students.
An Example of Inbound/Outbound
Assessment (Leadership Development)
In 20XX, we conducted a senior leadership
development program in partnership with
University of Virginia-Darden
Ideally, we would have:
◦ Defined the learning outcomes we wanted from the
program
◦ Conducted an Inbound Assessment of the
participants before they started the course
◦ Conducted an Outbound Assessment of the
participants after they completed the course
◦ Compared Inbound and Outbound scores to then
directly evaluate the effectiveness of the learning
program.
An Example of Inbound/Outbound
Assessment (Technician Training)
ElectricalApprentice Program
4 Year program registered with DOL
◦ Define the learning outcomes we want from the
program
◦ Conduct an Inbound Assessment of the
participants before they start the course
 Can be used to identify who is in course as well
◦ Conduct periodic “mid-point” assessments to
determine levels of retention
 MSHA requirements as far as working solo
◦ Conduct an Outbound Assessment of the
participants after they complete the course
◦ Compare Inbound and Outbound scores to then
directly evaluate the effectiveness of the learning
program.
Considerations for Industry
Not necessarily applicable or practical for all training
programs; however, it may have merit for some, usually more
long-term training programs.
Should always be doing some sort of pre/post testing to
evaluate both the students results and the effectiveness of the
training.
Testing should be tied to the learning outcomes associated
with the course and/or program.
Need to identify and differentiate between program-level
(position-level) and course-level (task-level) Learning
Outcomes.
Evaluating Employee Training
Effectiveness: Applying
Academic Principles for
Program-Level Learning
Outcomes Assessment
John Lavrenz, Tom Kaness, & Olin
Oedekoven, Ph.D.

Thank you!
Evaluating Employee Training
Effectiveness: Applying Academic
Principles for Program-Level Learning
Outcomes Assessment
John Lavrenz, Tom Kaness, & Olin
Oedekoven, Ph.D.
PO Box 741
1001 S Douglas Hwy, Suite 160
Gillette, WY 82716
307-685-1555
1-877-260-1555
www.PeregrineAcademics.com
www.PeregrineLeadership.com