Training & Development

Evaluation
Objectives

 Why Evaluate?
 Trends
 Case Studies (National Security Agency- NSA)
 The Framework: Evaluation Levels, targets
 Linking Needs Assessments with Evaluation
 Putting it all together
 Evaluation in Telkom SA Centre for Learning
(Implementation guidelines)
 Conclusion
Why Evaluate?
• T&D Not an isolated activity in Business
• Economic implications should form part of T&D
(What is the value Add? How do we quantify T&D Value add?)
• T&D Managers/Practitioners need to take a Business
approach to T&D
We evaluate to:
• Ensure sustainable OPEX /CAPEX.
(Training budgets are normally cut first)
• Provide proper value proposition to Business Sponsors.
(Value, Value,Value)
• Accountability?
 Trends
Traditional: Or is it?
 Considered as an add on after completion of program
 Reactive in nature. Completed upon request from sponsor
 Most evaluation results consisted of end-of program questionnaires
 Lack of consistent approaches
Latest trends
 Organisations are developing/Implementing Evaluation Frameworks
 Evaluations are designed into new programs and processes
 Systematic processes are used to measure the impact of interventions
 Roles and responsibilities are defined for Evaluation and Accountability
 Implementation and communication of the Evaluation process to Business
Sponsors
Case Studies
Various were discussed. Frameworks from over 100 Companies
were collated in the literature. All had similarities with the 4 Level
approach of Donald L. KirkPatrick and the 5th level ROI.
 Anderson Consulting
 Bell Atlantic Network Services
 Nortel (Northern Telecom)
 IIIinois Power Company? ( Electric and Gas utility)
NSA (National Security Agency)
• Embarked on a results based approach to T&D 1997
• Certified 50 trainers on ROI, Educated over 150 in ROI Principles
• Selected as best practice in 1999 for measuring Impact to learning
• Evaluation framework: KirkPatrick’s 4 Levels,
Phillips 5th Level for ROI
Evaluation Framework
Donald L. KirkPatrick’s model
• Probably the most admired, consistent and effective……
(Case Studies in “Implementing Evaluation Systems and Processes” Jack J Phillips)
The four Levels
• Level 1 – Reaction and Planned Action
Measures participants reaction and outlines specific plans for
Implementation
Level 2 – Learning
Measures skills knowledge or attitude changes
• Level 3 – Application and implementation
Measures changes in behaviour on the job and specific application
and implementation
• Level 4 – Business Impact
Measures Business impact of the program
• Level 5 – Return on investment
Compares the monetary value of the results with the cost of the program, expressed as
a%
Linking Needs Assessment to Evaluation

Needs Assessment Program Objectives Evaluation

ROI ROI
51 Potential Payoff

Impact Objectives Business Impact

41 Business Needs

31 Job Performance Needs Application Objectives Application

21 Skills/Knowledge Needs Learning Objectives Learning

11 Preferences Satisfaction Objectives Reaction

Evaluation Targets
 Refers to the percentage of HRD Programmes measured at a
specific level
Evaluation targets for a Large Telecommunication Company

Level % Of Courses

Level 1 - Reaction 100%

Level 2 - Learning 60%
Level 3 - Application 30%
Level 4 - Business Impact 10%
Level 5 - Return on Investment 5%

The setting of targets depends on the type of programmes, budgets allocation, resource
Availability etc.
Putting it all together (Jack J. Phillips)
Data Analysis

Tabulate
Costs of
programme
Planning Data Collection Reporting

Convert
Develop
Business Data
Evaluation Reaction L1 Learning L2 Application L3 Generate report
Plans and
Impact L4 to Monetary ROI L5
Value and present
Baseline Data
to Stakeholders

Identify
Isolate the effects Intangible benefits
of the programme
Evaluation in Telkom SA Centre for Learning

 Framework established, KirkPatrick’s 4 Level’s

 Level 1 - 100% of offerings, reports generated via SAP
 Level 2 - 70%, Data collection differs from program to program
(Eg. Performance tests, role plays, QMP’s etc.)
 Level 3 - 10% (Observations, Competency based Assessments,
action planning, performance contracting)
 Level 4 - 2% (Questionnaires, Performance monitoring)
 Level 5 - ???%
Implementation guidelines

 Establish an Evaluation policy

 Assign responsibilities
 Establish an Evaluation Framework
 Determine Evaluation targets
 Develop tools (L1-L5), systems processes, guidelines etc.
 Implement. ROI projects.
Conclusion

As HR Practitioners we need to:

“Provide higher margin people development solutions in an overly
commodotised training environment and capture the sustainable
bottom-line benefits of a solution-based people development
approach”.
What Questions do you have?