Session Number: Workshop 8

June 2, 2007
9:00 a.m. – 5:00 p.m.

Show Me the Money: Measuring ROI for

People, Projects, and Programs

Learning Objectives
• Make the business case for "showing the money" for a few
select programs
• Identify the steps, guiding principles, and issues involved in
"showing the money"
• Use impact and ROI data to drive improvement and change

Jack J. Phillips, Ph.D.

Phone (205) 678-8101 Fax (205) 678-8102

www.roiinstitute.net

© 2007 ROI Institute, Inc.

1
 The Value Evolution
Scope
“Show me the money!” We have heard it before, sometimes as a serious request and
other times as a half-hearted appeal to “prove it or go away.” Either way, it represents
the newest value statement. These days the concept of value of projects, programs,
and processes has changed dramatically. Consider the following:
• Unilever seeks a financial return on its investment in the arts.
• The US Air Force has developed the return on investment in data security to
prevent intrusion into its databases.
• Eskom, a large electric utility based in South Africa is interested in the financial
impact of a major transition program in its transmission division.
• The BBC in London is interested in knowing the financial impact of its major
leadership program.
• Businessweek.com has developed an ROI award for Hollywood’s big earners
comparing the net benefit divided by the cost of the investment in actors.
• Apple Computer has calculated the ROI for investing in process improvement
teams.
• Sprint/NEXTEL has developed the ROI on its diversity program.
• The Australian Capital Territory Community Care agency forecast the ROI for the
implementation of a client relationship management (CRM) system.
• GlaxoSmithKline is interested in showing the value of providing faster internet
connections for its pharmaceutical sales representatives.
• Accenture has calculated the ROI on a new sales platform for its consultants.
• Wachovia has developed the forecast and actual ROI for its negotiations
program.
• A major hotel chain has calculated the financial value and ROI of its coaching
program.
• The cities of New York, San Francisco, and Phoenix are showing the monetary
value of investing in projects and programs to reduce the number of homeless
citizens on the streets.
• The US Navy has developed the financial ROI on a succession program for top
leaders.
• Cisco Systems is measuring the ROI for its key meetings and events.
• A major US Defense Department agency has developed the ROI for a masters
degree program offered by a major university.
Questions for Discussion:
1. Any surprises here?
____________________________________________________________________
____________________________________________________________________

2. How do you define/describe value?

____________________________________________________________________
____________________________________________________________________
2
 The Many Applications of the ROI Methodology

Communications Policies/Procedures
Consulting/OD Procurement/Supply Chain
Educational Systems Public Policy Programs
Human Resources/Human Public Relations / Public Affairs
Capital
Leadership and Coaching Quality / Six Sigma
Learning and Development Research and Development
Marketing/Advertising Social Programs
Meetings and Events Technology / Systems

The “New” Definition of Value

Value Must:

ƒ Be balanced, with qualitative and quantitative data

ƒ Contain financial and non-financial perspectives
ƒ Reflect strategic and tactical issues
ƒ Represent different time frames
ƒ Satisfy all key stakeholders
ƒ Be consistent in collection and analysis
ƒ Be grounded in conservative standards
ƒ Come from credible sources
ƒ Reflect efficiency in its development
ƒ Create a call for action

Any surprises here?

Can you add to the list?

3
 The Changing Face of Value
Old
Activity cost center
Expense control
Human Resources as expenses
Rule centered
We value what IBM and GE value
Tolerate overhead
Add value with small pieces
Most of us are in a support role
Our measures are based on
benchmarking
We view value from one perspective
Just another days work in the cubicle
A job
Treat old ideas as new ideas
New
Results profit center
Maximize Value
Human capital as investment
Client centered
Our value systems are unique
Outsource or automate overhead
Add value with integrated applications
All of us are in a sales role
Our measures are based on what we
need
We view value from everyone’s
perspective
We do something of value
A performance
Treat old ideas as old ideas

Can you site examples of this?

The “Show Me” Evolution

Term Issue
Show Me! Collect Impact Data

Show Me the Money! And Convert Data to Money

Show Me the Real Money! And Isolate the Effects of the Project

Show Me the Real Money, And Compare the Money

And Make me Believe it! to the Cost of the Project

Have you experienced this evolution? Explain

4
 How “Show Me the Money” Connects

Organizational
Measures
(Sponsor)

Personal
Financial
Measures
Measures
(CFO) (Stakeholders)

SHOW
ME
THE
MONEY!!

Varieties of Terms and Applications

Name Example
Program A leadership development process for senior executives
Project A re-engineering project for the plastics division
System A fully networked system for all branches
Initiative A faith-based initiative to reduce recidivism
Policy A new pre-school policy for the disadvantaged
Procedure A new scheduling procedure for truck drivers
Event A golf outing for customers
Meeting US Coast Guard innovations conference
Process Quality sampling process
People Staff additions in the customer care center
Tool A new selection tool for the hotel staff

5
 The Results
Consider your most important program (i.e., a strategic, expensive, high-profile project
that attracts management attention). Suppose you conducted an impact study to
measure the success of the program. You discover that three months after completing
the program, participants have:
• reacted positively to the program and found it to be relevant to their work;
• learned new skills and gained new information;
• applied the skills and information routinely on the job, although they had some
difficulty in a few areas;
• improved several important work unit measures, with some measures improving
as much as 30%;
• achieved an impressive 105% return on investment; and
• reported an increase in job satisfaction in the work unit.

Questions for discussion:

1. What issues or concerns do you have about the data?

2. What would these types of data mean for the program?

3. What would the results mean to you personally?

4. If the above items were negative, what would it mean for the program?

5. What would negative results mean to you personally?

6. How should the results be used?

6
 Key Issues with This Level of Analysis
• Objectives?
• Credibility of data?
• Source of data?
• Consistent methodology?
• Scope?
• Standards?
• Use of data?
• Cost of process?
• Fear of data?

Project or Program Value Chain

Level Measurement Focus

0. Input Measures input such as volume and
efficiencies

1. Reaction & Planned Measures participant reaction to the program

Action and captures planned actions

2. Learning Measures changes in knowledge, skills, and

attitudes

3. Application Measures changes in on-the-job behavior or

actions and progress with actions

4. Business Impact Captures changes in business impact

measures

5. ROI Compares program benefits to the costs

Global Trends in Measurement and Evaluation*

ƒ Organizations are moving up the value chain in their evaluation strategies

ƒ Investment is increasing to 3-5% of the budget
ƒ Increased focus is driven by clients and sponsors
ƒ ROI is the fastest growing metric
ƒ Evaluation data is used to drive improvement and secure funding
ƒ Evaluation is addressed early and often in the implementation cycle
ƒ Processes are systematic and methodical, often designed into the delivery and
implementation processes
ƒ Technology is significantly enhancing processing
*Based on benchmarking with over 200+ organizations using a comprehensive
measurement and evaluation process.
7
 Activity Based Results Based
• No business need for the • Program linked to specific business
program
• No assessment of • Assessment of performance
performance issues effectiveness
Shifting Paradigms

• No specific measurable • Specific objectives for behavior and

objectives
• No effort to prepare program
participants to achieve results
• No effort to prepare the work
environment to support
transfer
• No efforts to build
partnerships with key
managers
• No measurement of results or
benefit-cost analysis
• Planning and reporting is input
focused
business impact
• Results expectations communicated
to participants
• Environment prepared to support
transfer
• Partnerships established with key
managers and clients
• Measurement of results and benefit-
cost analysis
• Planning and reporting is outcome
focused

Why does ROI work?

• Provides a balanced set of measures

• Offers a step-by-step process
• Bridges business evaluation and program evaluation
• Balances research/statistical methods with practical application
• Flexible for all types of programs
• Credible with managers and administrators

8
 Reliance Insurance Company
At the end of a monthly staff meeting, Frank Thomas, CEO of Reliance Insurance
Company, asked Marge Thompson, Manager of Training and Development, about the
Communications Workshops that had been conducted with all supervisors and
managers throughout the company. The workshop featured the Myers-Briggs Type
Indicator (MBTI) and showed participants how to interact with, and understand, each
other in their routine activities. The MBTI categorizes individuals into one of 16
personality types.

Frank continued, “I found the workshop very interesting and intriguing. I can certainly
identify with my particular personality type, but I am curious about what specific value
these workshops have brought to the company. Do you have any way of showing the
results of all 25 workshops?” Marge quickly replied, “We certainly have improved
teamwork and communications throughout the company. I’ve heard people make
comments about how useful the process has been to them personally.” Frank added,
“Do we have anything more specific? Also, do you know how much we spent on these
workshops?” Marge quickly responded by saying, “I’m not sure that we have any
precise data and I’m not sure exactly how much money we spent, but I can certainly find
out.” Frank concluded with some encouragement, “Any specifics would be helpful.
Please understand that I am not opposed to this training effort. However, when we
initiate these types of programs, we need to make sure they’re adding value to the
company’s bottom line. Let me know your thoughts on this issue in about two weeks.”

Marge was a little concerned about this CEO’s comments, particularly since he enjoyed
the workshop and made several positive comments about it. Why was he questioning
the effectiveness of it? Why was he concerned about the costs?

These questions began to frustrate Marge as she reflected over the past year and a half
in which every manager and supervisor had attended the workshop. She recalled how
she was first introduced to the MBTI. She attended a workshop conducted by a friend,
was impressed with the instrument, and found it to be helpful as she learned more
about her own personality type.

Marge thought the process would be useful to Reliance managers and asked the
consultant to conduct a session internally with a group of mid-level managers. With
favorable reaction, she decided to try a second group with the top executives, including
Frank Thomas. Their reaction was favorable. Then she launched it with the entire staff,
using positive comments from the CEO, and the feedback has been excellent.

With over 600 managers attending, she realized the workshops have been expensive.
She believes teamwork has improved, although there is no way of knowing for sure.
With some types of training you never know if it works, she thought. Still, Marge was
facing a dilemma. Should she respond to the CEO or just ignore the issue?

9
Questions for discussion:

1. Is this situation typical? Explain.

2. What are the basic issues in this case?

3. How could this situation be avoided in the future?

4. What should Marge do? Be specific.

5. What should Frank do if Marge does not respond?

10
 Is Your Organization A Candidate for ROI Implementation?
Check the most appropriate level of agreement for each statement:
1 = Strongly Disagree; 5 = Strongly Agree
Disagree Agree
1 2 3 4 5
1. My organization is considered a large organization with a wide † † † † †
variety of programs.

2. We have a large budget that attracts the interest of senior management. † † † † †

3. Our organization has a culture of measurement and is focused on

establishing a variety of measures in all functions and departments. † † † † †

4. My organization is undergoing significant change. † † † † †

5. There is pressure from senior management to measure results of our

programs. † † † † †

6. My function currently has a very low investment in measurement and

evaluation. † † † † †

7. My organization has experienced more than one program disaster in

the past. † † † † †

8. My department has a new leader. † † † † †

9. My team would like to be the leaders in our field. † † † † †

10. The image of our department is less than satisfactory. † † † † †

11. My clients are demanding that our processes show bottom-line results. † † † † †

12. My function competes with other functions with our organization for
resources. † † † † †

13. There is increased focus on linking our process to the strategic

direction of the organization. † † † † †

14. My function is a key player in change initiatives currently taking place

in the organization. † † † † †

15. Our overall budget is growing and we are required to prove the bottom
line of value of our processes. † † † † †

11
 Scoring
If you scored:
15 – 30 You are not yet a candidate for ROI.
31 – 45 You are not a strong candidate for ROI, however, it is time to start
pursuing some type of measurement process.
46 – 60 You are a candidate for building skills to implement the ROI process. At
this point there is no real pressure to show the ROI, which is the perfect
opportunity to perfect the process within the organization.
61 – 75 You should already be implementing a comprehensive measurement and
evaluation process, including ROI.
Are you a candidate?

• Accenture • Georgia Pacific

Private Sector Organizations using

• Allstate Insurance • Guthrie Healthcare Systems

• Apple Computer • Hewlett-Packard
• AT&T • Intel
• Bank of America • Lockheed Martin
• Banner Healthcare • Memorial Foundation Hospital
• Baptist Medical Center • Menlo Logistics
• Bristol-Myers Squibb • Microsoft
ROI

• Children’s Hospital of Los Angeles • Motorola

• Caremark RX • NCR
• Covenant Healthcare Systems • Nextel
• Dell Computers • PricewaterhouseCoopers
• Deloitte & Touche • QUALCOMM
• Delta Airlines • Shell Oil
• DHL Worldwide Express • UPS
• Federal Express • Wachovia Bank
• Genentech • Wal-Mart
Organizations using ROI

• State of Mississippi
• US Department of Defense
• State of Texas
• US Department of Navy
Public Sector

• State of New York

• US Department of Labor
• Government of New Zealand
• US National Security Agency
• Government of Singapore
• Central Intelligence Agency
• Government of Italy
• US Department of Veteran’s Affairs
• Government of Australia
• NASA
• Government of Canada

12
 Generates Six Types of Measures
• Reaction, Satisfaction, and Planned Actions
• Learning
• Application and Implementation
• Business Impact
• Return on Investment
• Intangible Measures
The ROI Methodology

….and includes a technique to isolate the effects of the program or solution

Grounded in Research and Application

• Approximately 5,000 impact studies are conducted each year
• Over 20,000 individuals have attended a two-day ROI workshop
• Over 4,000 individuals are certified to implement the ROI methodology
• The ROI methodology has been adopted by hundreds or organizations in
manufacturing, service, non-profit, and government settings in 44
countries
• The process has been refined over a 20-year period
• 16 books have been developed to support the process
• ROI Network has 600 members
• Several ROI Network conferences are conducted annually
• Ongoing research on the ROI methodology is conducted by the ROI
Institute to support practitioners

The ROI Calculation

Program Benefits
Benefit Cost Ratio (BCR) =
Program Costs

Net Program Benefits

ROI = X 100
Program Costs

>> Notes

13
 Measurement of Projects

Current Recommended
Status Comments
Level Measurement Category About
Status
Coverage Coverage

O Inputs/Indicators
Measures inputs into projects This is being
including number of projects, 100% 100% accomplished
audience, costs, and now.
efficiencies.

1 Reaction and Perceived Value Need more

Measure reaction to, and focus on
satisfaction with, the medium, 80-100% content and
content, and value of the project perceived
or program. value.
2 Learning and Confidence
Measures what participants Must use
understand or learned from the simple
project or program – 50-60%
learning
information, knowledge, skills, measures.
and contacts (take-aways).

3 Application and
Implementation
Measures progress after the Need more
program implemented – the use 15-25%
follow-up.
of information, knowledge, skills,
and contacts.

4 Impact and Consequences

Captures changes in business This is the
impact measures such as connection to
10%
output, quality, time, and cost- business
linked to the project or program. impact.

5 ROI
Compares the monetary
benefits of the business impact The ultimate
5%
measures to the costs of the evaluation.
project.

14
 Five Levels of Measurement - Examples
Level 0 Input and Indicators Level 3 Application and Implementation
• Number of projects • Use of Information
• Audiences • Use of Knowledge
• Web site hits • Use of Skill
• Request • Completion of Actions
• Attendance • Completion of Tasks
• Costs • Implementation of Ideas
• Time to Deliver • Following the Policy
• Use of Procedure
Level 1 Reaction and Perceived Value • Use of Regulation
• Relevance • Success with Application
• Importance • Barriers
• Usefulness • Enablers
• Appeal
• Emotion Level 4 Impact and Consequences
• Brevity • Productivity
• Uniqueness • Quality
• Concreteness • Incidents
• New Information • Efficiency
• Motivation • Compliance Discrepancies
• Appropriateness • Costs
• Intent to Use • Employee Engagement
• Employee Retention
Level 2 Learning and Confidence • Customer Service
• Information • Sales
• Knowledge • Customer Satisfaction
• Understanding Intangible Measures
• Capability …. includes a technique to isolate the effects of
• Contacts the communication project.
• Confidence
• Perceptions Level 5 Return on Investment
• Skills • ROI (%)
• Benefit Cost Ratio
• Payback Period

0 1 2 3 4 5

Input React Learn Apply Impact ROI

15
 Evaluation is like a Puzzle

An Evaluation Case Applications

Framework and Practice

Implementation

Operating
A Process Standards and
Model Philosophy

The purpose of each piece of the evaluation puzzle:

• Evaluation Framework

• Process Model

• Operating Standards

• Case Application

• Implementation

16
 The ROI Methodology
Capture Costs
Of Solution

Data Collection Data Analysis Reporting

LEVEL 1: LEVEL 3: Application
Evaluation Reaction and Planned and
Action Implementation
Planning
Develop/ Develop Collect Data Collect Data
Review Evaluation Isolate the Convert Data Calculate Generate
During After
Objectives of Plans and Solution Effects of to Monetary the Return On Impact
Solution
Solution Baseline Data Implementation Solution Value Investment Study
Implementation

LEVEL 2: LEVEL 4: LEVEL 5: ROI

Learning and Business Impact
Confidence

Identify
Intangibles

Intangible Benefits

ROI Calculation Evaluation Framework

Level Measurement Focus
Net Project Benefits
ROI = 1. Reaction & Planned Measure participant satisfaction with the
Project Costs
Action project/project and captures planned action.
Cost of project $230,000 2. Learning & Measures changes in knowledge, skills, and
Benefits of project (1st year) $430,000 Confidence attitude.
3. Application & Measures implementation and changes in
$430,000-$230,000 Implementation behavior in the performance setting.
ROI = = 087 x 100 = 87%
$230,000
4. Business Impact Measures changes in business impact variables.
5. Return on Compares benefits to the costs.
Investment

17
 12 Guiding Principles
1. When conducting a higher-level evaluation, collect
data at lower levels.
2. When planning a higher level evaluation, the
previous level of evaluation is not required to be
comprehensive.
3. When collecting and analyzing data, use only the most
credible sources.
4. When analyzing data, select the most conservative
alternatives for calculations.
5. Use at least one method to isolate the effects of the
program or project.
6. If no improvement data are available for a population or
from a specific source, assume that little or no
improvement has occurred.
7. Adjust estimates of improvements for the potential error
of the estimates.
8. Avoid use of extreme data items and unsupported claims
when calculating ROI calculations.
9. Use only the first year of annual benefits in the ROI
analysis of short-term solutions.
10. Fully load all costs of the solution, project, or program
when analyzing ROI.
11. Intangible measures are defined as measures that are
purposely not converted to monetary values.
12. Communicate the results of the ROI Methodology to all
key stakeholders.
The ROI Process
A comprehensive measurement and evaluation process
that generates six types of measures:
¾ Reaction and Planned Action
¾ Learning and Confidence
¾ Application and Implementation
¾ Business Impact
¾ Return on Investment
¾ Intangible Measures
This balanced approach to measurement includes a
technique to isolate the effect of the program or solution.
Results-based Solutions
¾ Performance solutions/projects are initiated,
developed and delivered with the end in mind.
¾ Participants understand their responsibility to
obtain results with programs/solutions.
¾ Support groups (management, supervisors, co-
workers, etc.) help to achieve results from
performance solutions.
¾ A comprehensive measurement and evaluation
system is in place for each program/project.
¾ Variety of approaches utilized to measure
contribution, representing a balanced viewpoint.
¾ Follow-up evaluations (Application, Impact, and
ROI) are developed for targeted solutions/projects
and results are reported to a variety of
stakeholders.
18
 Notes
Reaction & Planned Action

Learning

Application & Implementation

Isolate the Effects of the Program

Impact

ROI
Chain of Impact

Intangible Benefits

Multiple Stakeholders

Reaction & Planned Action

Learning

Application & Implementation

Stakeholder 2 Organization Stakeholder 3

Impact Impact Impact

ROI ROI ROI

Intangibles Intangibles Intangibles

Characteristics of Evaluation Levels

Power to
Chain of Value of Show Frequency Difficulty of
Impact Information Focus Results of Use Assessment

Lowest Consumer Lowest Frequent Easy

Reaction
Learning
Application
Impact
ROI
Highest Client Highest Infrequent Difficult

Consumers: The customers who are actively involved in the process.

Clients: The customers who fund, support, and approve the project.
19
 Cost-benefit Level 5
comparison

Evaluation Changes in
Purpose Level 4
outcome measures
Program
Need
Changes in
Level 3
Program performance
Profile
Stakeholder
Need for skills or
Needs Level 2
knowledge

Preferences Level 1

Guiding Principle #1

Guiding Principle #2

When Selecting Programs for Level 4 and Level 5 Evaluation,

Consider the Following
• Life cycle of the program
• Linkage of program to operational goals and issues
• Importance of program to strategic objectives
• Top administrator interest in the evaluation
• Cost of the program
• Visibility of the program
• Size of target audience
• Investment of time required

20
 Percent of Courses
Evaluation Targets
Level Current Target Suggested
Reaction 90 - 100%
Learning 40 – 60%
Application (Behavior) 30%
Impact 10 - 20%
Return on Investment 5 - 10%

• Executive Coaching • Competency Systems

• Leadership • Career Development Programs
ROI Application

• Diversity Programs • Recruiting Strategies

• Wellness/Fitness Initiatives
• Total Quality Management
• Self-Directed Teams
• Skill-Based/Knowledge-Based
Compensation
• Organization Development
• Meeting Planning
• Orientation Systems
• Associate Relations Programs
• Gainsharing Programs
• Technology Implementation
• e-Learning
• Safety & Health Programs
• Project Management

21
 Matching Evaluation Levels with Objectives

Instructions: For each objective listed below, indicate the level of evaluation at which
the objective is aimed.

1. Reaction 4. Business Impact

2. Learning 5. Return on Investment
3. Application
Evaluation
Objective Level
After completing this program or project, participants should:
1. Improve work group productivity by 20%
2. Initiate at least three cost reduction projects in 15 days.
3. Achieve an average cost reduction of $20,000 per project.
4. Increase the use of counseling discussion skills in 90% of situations where work
habits are unacceptable.
5. Achieve a 2:1 benefit to cost ratio one year after the new gainsharing program is
implemented.
6. Develop an understanding of how the employee assistance program works.
7. Increase the client satisfaction index by 25% in 3 months.
8. Integrate patient records into new database following the 5-step process.
9. Increase research grant dollars 20%.
10. Achieve a leadership simulation score average of 75 out of a possible 100.
11. Conduct a performance review meeting with direct reports to establish
performance improvement goals.
12. Receive a 4 out of 5 rating on appropriateness of new ethics policy.
13. Decrease the time to recruit new engineers from 35 days to 20 days.
14. Complete all items on the action plans in 3 months.
15. Achieve a positive reaction to flextime work schedule system.
16. Be involved in career enhancement program at a rate of 15%.
17. Decrease the number of security breaches of patient records.
18. Achieve a post-test score increase of 30% over pre-test.
19. Utilize new software daily as reflected by an 80% score on an unscheduled audit
of use.
20. Submit ideas or suggestions for improvement in the first year (10% target).

22
 Program Objectives Provide

• Direction to designers and developers

• Guidance to instructors and facilitators
• Goals for participants
• Satisfaction for program sponsors
• A framework for evaluators

Do you present Level 3 and 4 objectives to participants? ___________________

Needs Program
Linking Needs Assessment with Evaluation

Assessment Objectives Evaluation

Payoff ROI ROI
5 Needs Objectives 5

Business Impact Business

4 Needs Objectives Impact 4

3 Job Performance Application Application

Needs Objectives 3

2 Skills/Knowledge Learning Learning

Needs Objectives 2

Preference Satisfaction Reaction

1 Needs Objectives 1

23
 Linking Needs Assessment with Evaluation

Absenteeism is costing ROI of at least 25% Calculate the ROI

$100,000 month

Unexpected absenteeism Weekly absenteeism Monitor absenteeism data

problem exists rate will reduce by 10% six months for six months
after course

Dialogue between team Dialogue takes place, in 95% of Follow-up questionnaire to

leader/supervisor is not situations when an unexpected participants to check
occurring when there is an absence occurs. Environment frequency of skill application
unexpected absence supports safe dialogue. Identify barriers to dialogue -
three months post program

Deficiency in Acquisition of dialogue Skill practice sessions

counseling/discussion skills skills are demonstrated during program

Two-day communication skills Program receives favorable rating Reaction questionnaire

workshop must provide usable of 4 out of 5 on relevance; at the end of program
and relevant skills; facilitator- participants identify three planned
led; participants include actions
supervisors and team members

24
 Key Alignment Questions

Is this a problem worth What is the actual ROI?

solving?
What is the BCR?
Is there a potential pay
off? Needs Program
Assessment Objectives Evaluation Which business measure
improved?
What is the specific 5 Potential ROI ROI 5
measure? Payoffs Objectives How much is related to
the program?
What happens if we do
4 Business Impact Business 4
nothing?
Needs Objectives Impact
What has changed?
What is occurring or not 3 Job Performance Application Application 3 Which skills/knowledge
occurring on the job that Needs Objectives have been applied?
influences the business
measure?
2 Skills/Knowledge Learning Learning 2
Needs Objectives What did they learn?
What skills or knowledge Who did they meet?
is needed to support the Satisfaction
1 Preferences Objectives Reaction 1
job performance need?
What was the reaction to
the program?
How should the solution
be structured? Do we intend to
implement the program?

25
 Developing Reaction Objectives

Measuring Reaction and Satisfaction

Reaction objectives are critical in this measurement chain because they:

• Describe expected immediate and long term satisfaction
• Describe issues that are important to the success of the program
• Provide basis for evaluating the beginning of the measurement chain of impact
• Place emphasis on planned action, if feasible

The best reaction objectives:

• Identify issues that are important and measurable
• Are attitude-based, clearly worded, and specific
• Specify the participant has changed in thinking or perception as a result of the
program
• Underscores the linkage between attitude and the success of the program
• Represent a satisfaction index from key stakeholders
• Have the capability to predict program success

Key questions are:

• How relevant is this program?
• How important is this program?
• Are the facilitators effective?
• How appropriate is this program?
• Is this new information?
• Is this program rewarding?
• Will you implement this program?
• Will you use the concepts/advice?
• What would keep you from implementing objectives from this program?
• Would you recommend the program to others?

Examples of Level 1 Objectives

1.

2.

3.

26
 Developing Learning Objectives
Measuring Skills and Knowledge Enhancement

Learning objectives are critical to measuring learning because they:

• Communicate expected outcomes from instruction
• Describe competent performance that should be the result of training or learning
• Provide basis for evaluating learning
• Focus learning for participants

The best learning objectives:

• Describe behaviors that are observable and measurable
• Are outcome-based, clearly worded and specific
• Specify what the learner must do (not know or understand) as a result of the
training
• Have three components:

1. Performance—what the learner will be able to do at the end of the training

2. Condition—circumstances under which the learner will perform the task
3. Criteria—degree or level of proficiency that is necessary to perform the job

Three types of learning objectives are:

• Awareness — familiarity with terms, concepts, processes

• Knowledge — general understanding of concepts, processes, etc.
• Performance — able to demonstrate the skill (at least at a basic level)

Examples of Level 2 Objectives

1. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

2. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

3. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

27
 Developing Application Objectives

Measuring on the Job Application and Implementation

Application objectives are critical to measuring application of skills and knowledge
because they:
• Describe expected intermediate outcomes
• Describe competent performance that should be the result of the program
• Provide basis for evaluation of on the job performance changes
• Place emphasis on applying what was learned
The best application objectives:
• Identify behaviors that are observable and measurable
• Are outcome-based, clearly worded and specific
• Specify what the participant will change or has changed as a result of the
program
• May have three components:
1. Performance – what the participant has changed/accomplished at a specified
follow-up time after the program
2. Condition – circumstances under which the participant performed the task
3. Criteria – degree or level of proficiency under which the task or job was
performed
Two types of application objectives are:
• Knowledge based – general use of concepts, processes, etc.
• Behavior based – able to demonstrate the use of the skill (at least at a basic
level)
Key questions are:
• What new or improve knowledge will be applied on the job?
• What is the frequency of skill application?
• What new tasks will be performed?
• What new steps will be implemented?
• What new action items will be implemented?
• What new procedures will be implemented or changed?
• What new guidelines will be implemented or changed?
• What new processes will be implemented or changed?
Examples of Level 3 Objectives
1. ___________________________________________________________________
___________________________________________________________________

2. ___________________________________________________________________
___________________________________________________________________

3. ___________________________________________________________________
___________________________________________________________________
28
 Developing Impact Objectives
Measuring Business Impact

Impact objectives are critical to measuring business performance because they:

• Describe expected outcomes
• Describe business unit performance that should be the result of training or
instruction
• Provide basis for measuring the consequences of application of skills and
knowledge
• Place emphasis on achieving bottom line results

The best impact objectives:

• Must contain measures that are linked to the skills and knowledge taught in the
program
• Describe measures that are easily collected
• Are results-based, clearly worded and specific
• Specify what the participant has accomplished in the business unit as a result of
the training

Four types of impact objectives involving hard data are:

• Output focused
• Quality focused
• Cost focused
• Time focused

Three common types of impact objectives involving soft data are:

• Customer service focused
• Work climate focused
• Job satisfaction focused

Examples of Level 4 Objectives

1. ___________________________________________________________________
___________________________________________________________________

2. ___________________________________________________________________
___________________________________________________________________

3. ___________________________________________________________________
___________________________________________________________________

29
 Developing Impact Objectives

Business Impact from HR

Examples of Measures

OUTPUT TIME
Units Produced Equipment Downtime
Items Assembled Overtime
Revenue On Time Shipments
Items Sold Time to Project Completion
Forms Processed Processing Time
Inventory Turnover Cycle Time
Output Per Hour Supervisory Time
Productivity Training Time
Work Backlog Repair Time
Incentive Bonus Efficiency
Shipments Work Stoppages
New Accounts Generated Order Response
Late Reporting
COSTS Lost Time Days
Budget Variances
Unit Costs QUALITY
Cost by Account Scrap
Variable Costs Waste
Fixed Costs Rejects
Overhead Cost Error Rates
Operating Costs Rework
Number of Cost Reductions Shortages
Project Cost Savings Product Defects
Accident Costs Deviation From Standard
Sales Expense Product Failures
Inventory Adjustments
CUSTOMER SERVICE Number of Accidents
Customer Satisfaction Survey
Customer Satisfaction Index WORK CLIMATE
Customer Complaints Number of Grievances
Customer Comments Number of Discrimination Charges
Customer Defection Employee Complaints
Customer Retention Job Satisfaction
Organizational Commitment
WORK HABITS Employee Turnover
Absenteeism Reduced Litigation
Tardiness
First Aid Treatments
Violations of Safety Rules
Excessive Breaks

30
 Developing Level 3 and 4 Objectives

Think of a program that is linked to important organizational goals. Develop at least two
Level 3 and Level 4 objectives for the program. Make any assumptions you need to
complete the objectives.

Program Title:

Target Audience: Duration:

Level 3 Objectives:
After completing this program, participants will:

1. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

2. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

Level 4 Objectives:
After participants apply learned skills/behavior, their performance will impact:

1. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

2. ___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

31
 Option 1, When You Don’t Have a Clue
1. How did you use the material from this project or program?
2. What influence did it have in your work? Team?
3. What specific measure was influenced? Define it.
4. What is the unit value of the measure? (Profit or Cost)
5. What is the basis of this value?
6. How much did the measure change since the project was implemented?
7. What is the frequency of the measure? Daily, weekly, monthly, etc.
8. What is the total annual value of the improvement?
9. List the other factors that could have caused this total improvement?
10. What percent of the total improvement can be attributed to this project?
11. What is your confidence estimate, expressed as a percent, for the above data?
0% = no confidence; 100% = certainty

Option 2, When the Measure Is in a Defined Set

1. To what extent did this project or program positively influence the following
measures:

Significant No
Influence Influence
5 4 3 2 1 n/a
productivity € € € € € €
sales € € € € € €
quality € € € € € €
cost € € € € € €
efficiency € € € € € €
time € € € € € €
employee satisfaction € € € € € €
customer satisfaction € € € € € €
other € € € € € €

2. What other measures were positively influenced by this project?

3. Of the measures listed above, which one is most directly linked to the project?
(check only one)
ˆ productivity ˆ sales ˆ quality
ˆ cost ˆ efficiency ˆ time
ˆ employee satisfaction ˆ customer satisfaction ˆ other

4. Please define the measure above.

5. Indicate the specific unit of measurement.

32
 6. How much did this measure improve since you began this project?

7. What is the frequency of the measure? ˆ daily ˆ weekly ˆ monthly ˆ

annually
8. For this measure, what is the monetary value of improvement for one unit of this
measure? Although this is difficult, please make every effort to provide the value.

9. Please state your basis for the estimated value of one unit of improvement you
indicated above.

10. What is the total annual value of improvement in the measure you selected
above?

11. List the other factors that have caused this total annual improvement.

12. Recognizing that other factors may have caused this improvement, estimate the
percent of improvement related directly to this project of program?
%
13. What confidence do you place in the estimates you have provided in the prior
questions? (0% is no confidence, 100% is certainty.)
%

Option 3, When the Measure Is Known

1. Please define the first measure connected to your project.
2. Define the unit of measure.
3. For this measure, what is the monetary value of improvement for one unit of this
measure?
4. Please state your basis for the value of the unit of improvement you indicated above.
5. For the measure listed as most directly linked to the program, how much has this
measure improved in performance?
6. Indicate the frequency base for the measure. Daily, weekly, monthly, quarterly.
7. What is the annual value of improvement in the measure you selected above?
Multiply the increase (Question 5) by the frequency (Question 6) times the unit of
value (Question 4).
8. List the other factors that could have influenced these results.
9. Recognizing that the other factors could have influenced this annual value of
improvement, please estimate the percent of improvement that is attributable (or
isolated) to the program. Express as a percentage out of 100%.
10. What confidence do you place in the estimates you have provided in the questions
above? A 0% is no confidence, a 100% is certainty.

33
 Plan Your Project Evaluation

Program: _______________________________________________________
Evaluation Team: ________________________________________________
Expected Date of Completion: ______________________________________
1. What is your purpose in conducting an evaluation on this program?

2. What are the program objectives at each level of evaluation?

Level 1 ______________________________________________________
Level 2 ______________________________________________________
Level 3 ______________________________________________________
Level 4 ______________________________________________________
Level 5 ______________________________________________________

3. What are your measures of success for each objective?

Level 1 ______________________________________________________
______________________________________________________
Level 2 ______________________________________________________
______________________________________________________
Level 3 ______________________________________________________
______________________________________________________
Level 4 ______________________________________________________
______________________________________________________
Level 5 ______________________________________________________
______________________________________________________

34
 Data Collection Plan
Program:______________________________ Responsibility:_________________________ Date:____________
Data Collection
Level Broad Program Objective(s) Measures Method/Instruments Data Sources Timing Responsibilities

1
REACTION AND
PLANNED ACTIONS

2
LEARNING AND
CONFIDENCE

3
APPLICATION AND
IMPLEMENTATION

4
BUSINESS IMPACT

5 Comments:______________________________________________________________
ROI

______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________

35
 ROI Analysis Plan
Program:______________________________ Responsibility:_________________________ Date:____________

Methods for
Isolating the Methods of Other
Data Items Effects of the Converting Data Communication Influences/
(Usually Program/ to Monetary Cost Intangible Targets for Final Issues During
Level 4) Process Values Categories Benefits Report Application Comments

36
 The Methods: Level 3 Level 4
Collecting Post Program Data • Follow-up Surveys 9
• Follow-up Questionnaires 9 9
• Observation On the Job 9
• Interviews with Participants 9
• Follow-up Focus Groups 9
• Program Assignments 9 9
• Action Planning 9 9
• Performance Contracting 9 9
• Program Follow-up Sessions 9 9
• Performance Monitoring 9

When selecting data collection methods

• Type of data
• Time – Participant / Supervisor
Factors to Consider

• Costs
• Accuracy – Validity / Reliability
• Utility
• Culture / Philosophy
When determining timing of follow-up
• Availability of data
• Ideal time for behavior change (Level 3)
• Ideal time for business impact (Level 4)
• Convenience of collection
• Constraints on collection

• Use of a control group arrangement

Methods to Isolate

• Trend line analysis of performance data

Program Effects

• Use of forecasting methods of performance data

• Participant’s estimate of program impact (percent)
• Supervisor’s estimate of program impact (percent)
• Manager’s estimate of program impact
• Use of expert/previous studies
• Calculate/estimate the impact of other factors
• Customer input
37
 Applications of Data Collection Instruments
Matching Exercise
For each of the following situations, please indicate the most appropriate type of instrument
being used to collect data needed in the program’s evaluation. Select from these types:
A. Survey E. Focus Groups
B. Test F. Observation

Instrument
C. Questionnaire G. Performance Records
D. Interview

Level
Write the instrument’s letter in the box to the right of each question. Also, indicate
the level of evaluation pursued (1, 2, 3, or 4).

……
1. Customer service representatives have learned to resolve customer
complaints in the most effective manner. An integral part of the program
required customer service representatives to follow a series of planned steps
to resolve the complaint, using empathy and listening skills. As part of the
evaluation, the HR staff must determine the extent to which participants are
actually utilizing the newly acquired skills.

……
2. Intact team members are involved a conflict resolution program where they
acquired skills to resolve conflicts and disputes among themselves. Team
members in this work group have a high degree of interaction and some
responsibilities include checking the work of others. There had been an
unusually high level of friction with displays of open conflicts in the group. In
the program, participants learned how to deal with these issues and work
together as a smooth operating team. The HR staff needs to collect
information about the group’s progress, ideally in an environment where there
is an opportunity for group members to listen to comments from others.

……
3. Technicians participate in an e-learning program on basic mathematics and
are required to achieve a pre-determined level of competency in mathematics
after completing the program. The HR staff measures the level of
mathematical ability before and after the program.

……
4. The front desk staff at a major hotel has participated in a program to teach
them how to use a new reservation system that is being installed. As part of
the evaluation, it is important to obtain reactions to the program and capture
planned actions.

……
5. A company has implemented a new compensation plan in which the
employees share in the overall profits of the company. Employees have
attended a roll-out meeting where they have the opportunity to learn how the
program works and what is required of them to make it successful. As part of
the evaluation, management is interested in finding out what the employees
think about the new plan after attending the briefing.

……
6. Sales representatives have a new commission program designed to improve
sales. One objective of the program is to improve sales volume and the HR
staff must determine exactly what increase was achieved by each individual
since the program was conducted.

……
7. Supervisors attended a problem-solving program, where they learned a
logical approach to solving significant problems facing their work units. As a
part of the program’s evaluation, the HR staff needs feedback from
participants concerning their use of the acquired skills. The staff thinks there
is a possibility of a success story here and will need to probe for details.
38
 International Sales
(Part A)
You have recently completed an organization development project with the sales team
of a large international marketing group. The project was designed to increase sales
with existing customers, decrease the sale cycle, and reduce the cost of each sale.
Three months after the project is complete, all three measures have improved. In a
follow-up meeting with the VP of sales, you are presenting a summary of the project and
the improvements. Early in the presentation, the sales VP asks, “How much of this
improvement is related to your project?
Questions for discussion:

1. Can this question be appropriately addressed in the meeting?

2. Identify the range of possible techniques to isolate the affects of the project.

3. When should this issue be addressed?

4. Can the effects of a project always be isolated?

(Part B)
Questions for discussion:

1. Which technique(s) should be used to convert data to monetary value?

a. Increased sales
b. Decreased sales cycle time
c. Reduction of the cost of sale

2. When should this issue be addressed?

3. Should this conversion always be made?

39
 • What is the difference in improvement?
Experimental
M1 Program M2 Group

Control
M1 M2 Group
Control Group Method Design

• What is the difference?

Experimental
Program M1 Group

Control
M1 Group

• Does exposure to the measures make a difference?

Experimental
M1 Program M2 Group

Control
M1 M2 Group

Program M1 Experimental
Group (no pre-measure)

Example of Trend Line Analysis

1.85% Pre Program Six-Month
CPI Program Conducted
2% Projected Average
— Using Pre Data
as a Base 1.45%
ERROR
RATE

.7% Post
1% Program Six-
Month Average

J F M A M J J A S O N D J
MONTHS

40
 Isolating the Effects of a Program
Matching Exercise
Instructions: For each of the following situations, please indicate the method used to
isolate the effects of the program. Select from these methods:

A. Control group D. Participant’s estimate

B. Trend line analysis E. Use of customer input
C. Forecasting F. Expert’s estimate
Write the letter in the box to reflect the method used.

Situation Method
1. A manufacturing company has recently implemented a new incentive plan to boost
the sales for client partners. Just as the plan was implemented, the company
increased its promotional budget for each product line. Both the sales incentive plan

…
and the sales promotion have driven an increase in sales. It appears that no other
factors have contributed to this increase. There is a mathematical relationship
between the promotional budget and the sales increase based on historical data.
This equation is used to predict the sales increase based on the promotional budget
increase. This forecast is compared with actual figures to isolate the impact of the
sales incentive program.
2. Absenteeism for bus drivers in a large metropolitan area has been deteriorating for
some time. An HR program was implemented to include a no-fault absenteeism
policy and a change in the selection process. After the program was conducted, the
absenteeism rate decreased. It appears that no other influences have contributed to
this decrease. The pre-program absenteeism data are very stable and a trend is
projected in the post period to compare with the actual figures. The difference in the
…
two showed the contribution of the project.
3. An agent-training program in a real estate firm was designed to increase listings.
Customers are providing their rationale for deciding to list a home with a particular
agent. Listings increased three months after the program was completed. While
there are many factors that caused the increase, one factor was the quality of the
presentation made by the agent, which was the basis of the program. This
information was used to understand the impact of the agent-training program on the
…
actual number of houses listed.
4. A new wellness and fitness center has caused this energy company’s healthcare

…
expenditures to decrease one year after the center was opened. Given this amount
of decrease, several experts were assembled who understand why healthcare costs
have changed. These individuals are asked to explain all the contributing factors and
isolate the effects of the wellness and fitness center on that measure.
5. A large automobile company implemented a sales consulting process on a pilot basis.
Twelve dealerships were used in the initial pilot program. A comparison group was
selected to judge performance along several measures: sales volume, economy in
the market, the sales vs. service mix, incentives provided to sales staff, and the
quality rating of the dealership. The difference in the two groups showed the impact
…
of the sales consulting.

41
6. In a leadership development program for a biotech company, the participating

…
managers were asked to provide details on the impact of the use of the leadership
skills, using actual data in the work unit. As part of the exercise, these participants
estimated the percent of improvement directly related to the leadership development
program.

Adjusted
Example of Estimation

Factor that Percent of Confidence Percent of

Influenced Improvement Expressed as a Improvement
Improvement Caused By Percent Caused By
Program 60% 80% 48%
System Changes 15% 70% 10.5%
Environmental
Changes 5% 60% 3%
Compensation
Changes 20% 80% 16%
Other _______% _______% _______%
Total 100%

• Reputation of the source

Credibility of Data is

o Source of data
o Source of the study
Influenced by

• Biases
o Motives of the researcher
o Personal bias of the audience
• Methodology Used
o Assumptions made in the analysis
o Realism of the outcome data
o Type of data
• Scope of analysis

• Converting output to contribution – standard value

• Converting the cost of quality – standard value
Data are Converted by

• Converting employee’s time – standard value

• Using historical costs
• Using internal and external experts
• Using data from external databases
• Linking with other measures
• Using participants’ estimates
• Using supervisors’ and managers’ estimates
• Using staff estimates

42
 Cost of One Turnover from External Database
Example
Salary of Middle Manager $70,000/annually
Value of Turnover* 150% of annual salary
Cost of Turnover $105,000

*Value obtained from industry-related study (external data)

Cost of a Sexual Harassment Complaint using Historical Costs and

Expert Input

35 Complaints

Actual Costs from Additional Estimated Costs

Records from Staff

Legal Fees, Settlements, EEO/AA Staff Time,

Losses, Material, Direct Management Time
Expenses

$852,000 Annually

$852,000
Cost per complaint = $24,343
35

To Convert or Not Convert

• Is there a standard value?

• Is there a method to get there?
• Can we get there with minimum resources?
• Can we convince our executive in two minutes that the value is credible?

43
 Converting Data to Money
Matching Exercise
Instructions: For each of the following situations, please indicate the method used to
convert data to money. Select from these methods:
A. Profit/savings from output (standard value)
B. Cost of quality (standard value)
C. Employee time as compensation (standard value)
D. Historical costs/savings from records
E. Expert input
F. External database
G. Linking with other measures
H. Participant estimation
I. Management estimation
J. Estimation from HR staff
Write the letter in the box to reflect the method used.

Situation Method
1. The Veteran’s Administration was experiencing a high turnover rate of nurses. A
new HR program was designed to reduce this turnover. To obtain a value of one
voluntary turnover, the internet was used to find a study in healthcare that showed
the average cost (fully loaded) of replacing a nurse. This number, expressed as a
‰
percent of salary, was used in the calculation.
2. A new program for couriers at DHL was designed to reduce the number of re-
packaging errors. This error occurs when a package is damaged by the couriers
and has to be repackaged before delivery. The quality office in Brussels had
previously determined the standard cost for a repackaging error. This amount was
‰
used to develop the total monetary value for error reduction.
3. Mid-level managers at an electric utility were involved in a time management
program designed to help managers save time. Each manager estimated the
number of hours saved each week directly attributable to this program. The value
used for each hour saved was the total annual salary for those managers adjusted
‰
for employee benefits and calculated on an hourly basis.
4. An Australian government agency was implementing a new HR program designed
to reduce the number of stress claims for employees. The employees who worked
with angry and upset people were suffering from extreme stress on the job and
were filing claims. To obtain the monetary value for a stress claim, the medical ‰
and health staff provided an average value for one claim based on their expertise
of managing stress claims for several years.
5. A new customer call center program at a home appliance company was designed
to reduce the number of calls that were escalated to the next level of
management. The individuals involved in the program were the immediate
supervisors of the employees taking the calls. To determine the cost of a call ‰
escalation, the participants (supervisors) estimated the cost attached to each of
these calls.
44
6. A pharmaceutical company was implementing a new ethics program for all
employees. While there were several outcomes from this program, one in
particular measure was expense account violations. To obtain the average cost of
an expense account violation, all of the violations for a two-month period were
‰
taken directly from the records and divided by the total violations.
7. Employee engagement data are collected for a global computer company. A new
HR program was designed to improve engagement scores. To place a value on a
change in engagement score, the staff examined the correlations between
engagement scores and employee turnover in different job groups. This
correlation analysis is part of the human capital management system. As ‰
engagement scores improved, voluntary turnover decreased. The corresponding
cost savings of the turnover reduction was used as the value for changes in the
engagement score.
8. A small equipment manufacturing company was interested in reducing
absenteeism. The program was implemented and the cost of one absence was
needed for the monetary impact. The HR staff member who conducted the study ‰
estimated the cost to be $300 per day.
9. A major retail store chain was anxious to reduce the number of customer
complaints. A new program was implemented and the cost of a complaint was
needed. To obtain the cost of one complaint, the management of the customer
service area and the Customer Care Vice President estimated the average cost of
‰
one complaint.
10. Wachovia Bank implemented an advanced negotiation program where commercial
bankers increased revenue from new and existing clients. The outcome of the
program was increased revenue in specific product lines. To calculate the
monetary value of a sale, the revenue amount was multiplied by the profit margin ‰
for the product line, which was considered to be a standard value for the
organization.

45
5-Step Data Step 1: Focus on a unit of measure
Conversion
Step 2: Determine the value (V) of each unit

Step 3: Calculate the change in performance (∆P)

Step 4: Determine the annual amount of change (A∆P)

Step 5: Calculate the total annual value of the improvement (A∆P x V)

Step 1: One grievance

Internal Experts
Example using

Step 2: V = $6,500 (from Director of Nursing and HR experts)

Step 3: ∆P = average of 7 out of 10 grievances prevented per month
due to program

Step 4: Annual ∆P =

Step 5: A∆P x V =

Which cost category is appropriate for ROI?

A B
• Operating Costs • Administrative Costs
• Support Costs • Participant Compensation and
Facility Costs
• Classroom Costs
C D
• Program Development Costs • Analysis Costs
• Administrative Costs • Development Costs
• Classroom Costs • Delivery Costs
• Participant Costs • Overhead/Administrative Costs
• Evaluation Costs

>> Notes

46
 • Assessment costs (prorated)
• Development costs (prorated)
Fully-Loaded Cost

• Program materials
• Instructor/Facilitator costs
Profile

• Facilities costs
• Travel/Lodging/Meals
• Participant salaries and benefits
• Administrative/Overhead costs
• Evaluation costs

• Increased job satisfaction

• Increased organizational commitment
Intangible
Potential

Benefits

• Improved teamwork
• Improved customer service
• Reduced complaints
• Reduced conflicts
• Reduced stress

ROI is reported in one of two ways

Program Benefits
Benefit Cost Ratio (BCR) =
Program Costs

Net Program Benefits

ROI (%) = X 100
Program Costs

>> Notes

47
 Calculate the ROI

Costs per program (25 participants) – $80,000

Benefits per program (1st year) – $240,000

Benefit Cost Ratio (BCR) =

ROI = X 100 =

• Set the value as with other investments – 15%

acceptable
What is an

• Set the value slightly above other investments – 25%

ROI?

• Set at break even – 0%

• Set at client expectations

Evaluation Targets

ROI Level 5: 5-

Impact
Level 4: 10-

Level 3: 30%
Application

Level 2: 40-
Learning

Reaction and Planned Action Level 1: 90-100%

Criteria for Selecting Programs – Level 4 & 5 Evaluation

• Expected life cycle of the program
• The importance of the program in meeting organization’s goals
• The cost of the program
• Visibility of the program
• The size of the target audience
• Extent of management interest

48
What
When properly implemented, high ROI values can be achieved with
programs on:

• Leadership
• Team Building 100% to 700% ROI is not
• Management Development uncommon
• Supervisor Training
• Sales Training

ROI Best Practices

1. The ROI methodology is implemented as a process improvement tool and not a

performance evaluation tool for the learning/development staff.
2. ROI impact studies are conducted very selectively, usually involving 5-10% of
programs.
3. A variety of data collection methods are used in ROI analysis.
4. For a specific ROI evaluation, the effects of learning/development are isolated
from other influences.
5. Business impact data are converted to monetary values.
6. ROI evaluation targets are developed, showing the percent of programs
evaluated at each level.
7. The ROI methodology generates a micro level scorecard.
8. ROI methodology data are being integrated to create a macro scorecard for the
learning/development function.
9. The ROI methodology is being implemented for about 3-5% of the
learning/development budget.
10. ROI forecasting is being implemented routinely.
11. The ROI methodology is used as a tool to strengthen/improve the
learning/education process.

49
 Time/Cost Savings Tips
1. Plan for Evaluation –Early
• Plan the appropriate level of evaluation for each project
• Plan the data collection – who, what, when, how
• Plan the analysis
• Plan the communication of results
• Plan for the use of data

2. Build in Evaluation Tools

• Include questionnaires in handout
• Secure commitment during communication
• Develop action plan during communication
• Forecast values as an exercise

3. Share Responsibilities
• Use client/sponsors
• Use attendee/participants
• Use volunteers
• Use vendors/suppliers
• Use client staff

4. Communicate Expectations
• Describe in brochures
• Provide advance notice
• Announce requirements during communication
• Review follow-up questionnaires at the communication

5. Use Standard Tools and Templates

• Use standard reaction questionnaires (low, moderate, high intensity)
• Use standard learning questions
• Use standard application questionnaire
• Use standard impact questions
• Use standard reporting format

6. Use Estimates
• For forecasts
• For application on the job
• For impact
• For monetary values
• For costs

7. Use Shortcut Methods

• Questionnaires for data
• Estimates for isolation
• Use participants as only source
• Use small samples
• Use standard values

50
8. Use Sampling, Routinely
• Select only a few meetings for follow-up
• Select even fewer for ROI analysis
• Select only a sample of attendees

9. Streamline Reporting
• Create an executive summary
• Create a one-page summary
• Use the Web site
• Put it in a brochure

10. Develop Internal Capability

• 2-day workshop
• 5-day certification
• Books/Case Studies
• Coaching/Mentoring
• Join a network
• Create a network

11. Use Technology

• Online input
• Scanning technology
• Audience response system
• Survey software
• ROI software

51
 Healthcare, Inc.
Sexual Harassment Prevention Workshop

• Target Group: All supervisors and managers (655) with subsequent meetings with all
employees (6,844)

• Data Collection
(3) Self Assessment Questionnaire – 6 months after program
(3) Employee Survey (25% sample) – 6 months after program
(4) Complaint and Turnover Records – 12 months after program

• Isolating the Effects of the Program

Complaints – Trendline Analysis
Turnover – Forecasting

• Converting Data to Monetary Values

Complaints – Historical costs and input from experts (internal EEO/AA staff)
Turnover – External studies within industry

• Program Costs
Fully loaded to include needs assessment, development, coordination, participant
salaries and benefits, and evaluation
Total Costs = $277,987

• Intangible Benefits
Job Satisfaction, absenteeism, stress reduction, community image, and recruiting

• ROI Calculation
Monetary benefits from complaint reduction
Value of one internal complaint = $24,343
Annual improvement related to program = 14.8 complaints (prevented)
_______________________________________________

Monetary benefits from turnover reduction

Value of one turnover statistic = $20,887
Annual improvement related to program = 136 turnovers (prevented)
________________________________________________

Calculate the following:

Total Benefits
BCR = =
Program Costs

Total Benefits – Program Costs

ROI = X 100 =
Program Costs

52
 Why the concern?
• Measurement and evaluation are meaningless without communication
• Communication is necessary for making improvement
• Communication is a sensitive issue
Communication

• Different audiences need different information

Principles
• Keep communication timely
• Target communication to specific audiences
• Carefully select communication media
• Keep communication consistent with past practices
• Incorporate testimonials from influential individuals
• Consider the training function’s reputation when developing the overall
strategy

There are four types

1. Complete report
2. Executive summary
3. General audience summary
4. Streamlined report

The complete report includes the details

Reports

• General information
• Methodology for impact study
• Data analysis
• Costs
• Results
• Barriers and enablers
• Conclusions and recommendations
• Exhibits

53
 Sample Table of Contents for an ROI Impact Study

Table of Contents
List of Tables
List of Figures
List of Exhibits

Part I The Challenge and The Approach

Section 1: Introduction
Section 2: The Program
Section 3: Model for Impact Study
Section 4: Data Collection Strategy

Part II The Results

Section 5: Reaction and Satisfaction

Section 6: Learning
Section 7: Application and Implementation
Section 8: Business Impact
Section 9: Program Costs
Section 10: ROI and Its Meaning
Section 11: Intangible Benefits

Part III Recommendations

Section 12: Barriers and Enablers

Section 13: Suggestions for Improvement
Section 14: Conclusions
Section 15: Recommendations

54
 Sample Table of Contents for an Executive Summary

1. Introduction

2. The Training Program

3. Model for Impact Study

4. Data Collection Strategy

5. Measuring Reaction and Satisfaction

6. Measuring Learning

7. Measuring Application and Implementation

8. Measuring Business Impact

9. Intangible Benefits

10. Barriers and Enablers

11. Suggestions for Improvement

12. Conclusions

13. Recommendations

55
 Sprint/Nextel

Program Title: Diversity

Target Group: Managers and Employees
Solution: All-Inclusive Workforce Program (AIW)

RESULTS
Level 1: Level 2: Level 3:
Level 4: Level 5: Intangible
Reaction and Learning and Application and
Impact ROI Benefits
Planned Action Confidence Implementation
Composite Averaged Managers: Attrition Rate BCR: 2.6 Employee
Rating: 4.39 out 4.28 out of 5 Supports AIW Improvement Satisfaction
ROI:
of 5 (for six items) (for learning (87%) = 9.77%
163% Communication
on six
objectives) Addresses Cooperation
Problems (81%)
Diversity Mix
Encourages Staff
(78%) Teamwork
Employees:
Supports AIW
(65%)
Identifies
Differences (63%)
Encourages Staff
(60%)
91% of Managers
successful
completed action
plans

Technique to Isolate Effects of Program: Manager’s estimate, adjusted for error

Technique to Convert Data to Monetary Value: Standard cost item ($89,000 per
Turnover)
Fully-loaded Program Costs: $1,216,836

56
Building an ROI • Provides macro-level perspective of success

• Serves as a brief report versus detailed study

Scorecard

• Shows connection of training’s contribution to business objectives

• Integrates various types of data

• Demonstrates alignment between programs, strategic objectives,
and operating goals

0 Indicators / Scope / Volume

1 Reaction / Satisfaction
Types of Data

2 Learning
3 Application / Barriers / Enablers
4 Business Impact
5 ROI
Intangibles

Reporting for a Corporate University

0. Indicators
1. Number of employees involved
2. Total hours of involvement
3. Hours per employee
4. Training investment as a percent of payroll
5. Cost per participant

I. Reaction and Planned Action

1. Percent of programs evaluated at this level
2. Ratings on seven items vs. target
3. Percent with action plans
4. Percent with ROI forecast

II. Learning
1. Percent of programs evaluated at this level
2. Types of measurements
3. Self assessment ratings on three items vs. targets
4. Pre/Post – average differences

57
III. Application
1. Percent of programs evaluated at this level
2. Ratings on three items vs. targets
3. Percent of action plans complete
4. Barriers (list of top ten)
5. Enablers (list of top ten)
6. Management support profile

IV. Business Impact

1. Percentage of programs evaluated at this level
2. Linkage with measures (list of top ten)
3. Types of measurement techniques
4. Types of methods to isolate the effects of programs
5. Investment perception

V. ROI
1. Percent of programs evaluated at this level
2. ROI summary for each study
3. Methods of converting data to monetary values
4. Fully loaded cost per participant

Intangibles
1. List of intangibles (top ten)
2. How intangibles were captured

What barriers will prevent you from implementing ROI?

58
 ROI Quiz

True or False? Please choose the answer you feel is most correct

T F
1. The ROI Methodology generates just one data item, expressed as a
† †
percentage.
2. A program with monetary benefits of $200,000 and costs of $100,000
† †
translates into a 200% ROI.
3. The ROI Methodology is a tool to strengthen and improve the projects,
† †
programs, and processes.
4. After reviewing a detailed ROI impact study, senior executives will usually
† †
require ROI studies on all programs.
5. ROI studies should be conducted very selectively, usually involving 5-10%
† †
of programs.
6. While it may be a rough estimate, it is always possible to isolate the effects
† †
of a program on impact data
7. A program costing $100 per participant, designed to teach basic skills with
† †
job related software, is an ideal program for an ROI impact study.
8. Data can always be converted to monetary value, credibly. † †
9. The ROI Methodology contains too many complicated formulas. † †
10. The ROI Methodology can be implemented for about 3-5% of my HR/
† †
learning/development budget.
11. ROI is not future oriented; it only reflects past performance. † †
12. ROI is not possible for soft skills programs. † †
13. If an ROI impact study, conducted on an existing program, shows a
negative ROI, the client is usually already aware of the program’s lack of † †
results.
14. The best time to consider an ROI evaluation is three months after the
† †
program is completed.
15. In the early stages of implementation, the ROI Methodology is a process
improvement tool and not a performance evaluation tool for the HR or † †
learning/development staff.
16. If senior executives are not asking for ROI, there is no need to pursue the
† †
ROI Methodology.

59
 So, how did you do?

Now that the answers to the quiz have been explained, see how you fared. Tally your
scores. Based on the interpretations below, what is your ROI acumen?

No. of Correct
Responses Interpretation
14-16 You could be an ROI consultant
10-13 You could be a speaker at the next ROI Conference
7-9 You need a copy of a thick ROI book
4-6 You need to attend a two-day ROI workshop
1-3 You need to attend the ROI certification

60
 Increasing Questionnaire Response Rates
• Provide advance communication about the questionnaire.
• Clearly communicate the reason for the questionnaire.
• Indicate who will see the results of the questionnaire.
• Show how the data will be integrated with other data.
• Keep the questionnaire simple and as brief as possible.
• Keep questionnaire responses anonymous – or at least confidential.
• Make it easy to respond; include a self-addressed, stamped envelope/e-mail.
• Use the local manager to distribute the questionnaires, show support, and
encourage response.
• If appropriate, let the target audience know that they are part of a carefully
selected sample.
• Use one or two follow-up reminders.
• Have the introduction letter signed by a top executive.
• Enclose a giveaway item with the questionnaire (pen, money, etc.).
• Provide an incentive (or chance of incentive) for quick response.
• Send a summary of results to target audience.
• Distribute questionnaire to a captive audience.
• Consider an alternative distribution channel, such as e-mail.
• Have a third party gather and analyze data.
• Communicate the time limit for submitting responses.
• Consider paying for the time it takes to complete the questionnaire.
• Review the questionnaire at the end of the formal session.
• Carefully select the survey sample.
• Allow completion of the survey during work hours.
• Add emotional appeal.
• Design questionnaire to attract attention, with a professional format.
• Let participants know what actions will be taken with the data.
• Provide options to respond (paper, email, web-site).
• Use a local coordinator to help distribute and collect questionnaires.
• Frame questions so participants can respond appropriately and make the
questions relevant.

61
 The Wisdom of Crowds*
One day in the fall of 1906, British scientist Francis Galton left his home in the town of
Plymouth and headed for a country fair. Galton was eighty-five years old and beginning
to feel his age, but he was still brimming with the curiosity that had won him renown—
and notoriety—for his work on statistics and the science of heredity. On that particular
day, what Galton was curious about was livestock.

Galton’s destination was the annual West of England Fat Stock and Poultry Exhibition, a
regional fair where the local farmers and townspeople gathered to appraise the quality
of each other’s cattle, sheep, chickens, horses, and pigs. Wandering through rows of
stalls examining workhorses and prize hogs may have seemed a strange way for a
scientist to spend an afternoon, but there was certain logic to it. Galton was a man
obsessed with two things: the measurement of physical and mental qualities and
breeding. And what, after all, is a livestock show but a big showcase for the effects of
good and bad breeding?

Breeding mattered to Galton because he believed that only a very few people had the
characteristics necessary to keep societies healthy. He had devoted much of his career
to measuring those characteristics, in fact, in order to prove that the vast majority of
people did not have them. His experiments left him with little faith in the intelligence of
the average person, “the stupidity and wrong-headedness of many men and women
being so great as to be scarcely credible.” Galton believed, “Only if power and control
stayed in the hands of the select, well-bred few, could a society remain healthy and
strong.”

As he walked through the exhibition that day, Galton came across a weight-judging
competition. A fat ox had been selected and placed on display, and members of a
gathering crowd were lining up to place wagers on what the weight of the ox would be
after it had been slaughtered and dressed. For sixpence, an individual could buy a
stamped and numbered ticket, fill in their name, occupation, address, and estimate.
The best guesses would receive prizes.

Eight hundred people tried their luck. They were a diverse lot. Many of them were
butchers and farmers, who were presumably expert at judging the weight of livestock,
but there were also quite a few people who had no insider knowledge of cattle. “Many
non-experts competed,” Galton wrote later in the scientific journal Nature. “The average
competitor was probably as well fitted for making a just estimate of the dressed weight
of the ox, as an average voter is of judging the merits of most political issues on which
he votes.”

* Taken from The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective

Wisdom Shapes Business, Economics, Societies and Nations. James Surowicki. New York.
Doubleday, 2004

62
Galton was interested in figuring out what the “average voter” was capable of because
he wanted to prove that the average voter was capable of very little. So he turned the
competition into an impromptu experiment. When the contest was over and the prizes
had been awarded, Galton borrowed the tickets from the organizers and ran a series of
statistical tests on them. Galton arranged the guesses (totaling 787 – thirteen were
discarded because they were illegible) in order from highest to lowest and graphed
them to see if they would form a bell curve. Then, among other things, he added all the
contestants’ estimates, and calculated the mean of the group’s guesses. That number
represented, you could say, the collective wisdom of the Plymouth crowd. If the crowd
were a single person, that was how much it would have guessed the ox weighed.

Galton undoubtedly thought that the average guess of the group would be way off the
mark. After all, mix a few very smart people with some mediocre people and a lot of
dumb people, and it seems likely you’d end up with a dumb answer. But Galton was
wrong. The crowd had guessed that the ox, after it had been slaughtered and dressed,
would weigh 1,197 pounds. After it had been slaughtered and dressed, the ox weighed
1,198 pounds. In other words, the crowd’s judgment was essentially perfect. The
“experts” were not close. Perhaps breeding didn’t mean so much after all. Galton wrote
later: “The result seems more creditable to the trustworthiness of a democratic
judgment than it might have been expected.” That was, to say the least, an
understatement.

What Francis Galton stumbled on that day in Plymouth was the simple, but powerful,
truth: under the right circumstances, groups are remarkably intelligent, and are often
smarter than the smartest people in them. Groups do not need to be dominated by
exceptionally intelligent people in order to be smart. Even if most of the people within a
group are not especially well-informed or rational, they can still reach a collectively wise
decision.

Questions for discussion:

1. What implications does this concept have in evaluation?
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

2. Can you cite other examples?

___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

63
 ACTION PLAN
Name: Date:

Date to Date to
Potential Barriers to Implementing ROI Action Steps to Overcome these Barriers begin complete

Who do I need to help me achieve these actions?

What additional information or education will help me achieve these actions?

___________

65
 • Resources
o Funding
o Time
Barriers to ROI

o People
• Support
o Managers
o Colleagues
o Participants
• Skills
• Systems
• Culture

What actions can you take to overcome ROI barriers?

• Adapt versus adopt • Develop staff skills

Actions to Make ROI

• Planning and discipline o ROI Institute

o Establish goals and targets o International ROI Networks
o Assign responsibilities o Workshops
Work

• Improve needs analysis o ROI Certification

process o Competency groups
• Implement cost-savings o Read books / case studies / articles
approaches o Develop your own case study
• Communicate progress o Teach others

66
 The Basics
• ROI is the ultimate measure of profitability of our projects,
programs, and processes.
• Reporting the ROI metric alone is insufficient.
• The ROI process develops a balanced set of measures
representing a chain of impact.
• Not all programs should be evaluated to ROI.
The Barriers
• Resources
o Funding
Key Points to Remember

o Time
o People
• Support
o Managers
o Colleagues
o Participants
• Skills
Solutions to Barriers
• Adapt versus adopt
• Planning and discipline
• Improve needs analysis process
• Implement cost savings approaches
• Communicate progress
• Develop staff skills
The Benefits
• Show the contribution of programs
• Earn respect of senior management
• Gain the confidence of clients
• Improve support for training and performance improvement
• Enhance training and performance improvement processes
• Identify inefficient programs that need to be redesigned
• Identify successful programs

What if you do nothing?

67
 Sample of Published ROI Studies
Measuring the ROI: Key Impact Measures: ROI

Performance Management A variety of measures, such as productivity, quality, 298%1

(Restaurant Chain) time, costs, turnover, and absenteeism

Process Improvement Team Productivity and labor efficiency 182%1

(Apple Computer)

Skill-Based Pay Labor costs, turnover, absenteeism 805%2

(Construction Materials Firm)

Sexual Harassment Complaints, turnover, absenteeism, job satisfaction 1052%2

Prevention (Health Care
Chain)

Safety Incentive Plan (Steel Accident frequency rate, accident severity rates 379%2
Company)

Diversity (Nextel Retention, employee satisfaction 163%6

Communications)

Retention Improvement Turnover, staffing levels, employee satisfaction 258%3

(Financial Services)

Absenteeism Absenteeism, customer satisfaction 882%2

Control/Reduction Program

Stress Management Program Medical costs, turnover, absenteeism 320%2

(Electric Utility)

Executive Leadership Team projects, individual projects, retention 62%2

Development (Financial)

E-Learning (Petroleum) Sales 206%2

Internal Graduate Degree Retention, individual graduate projects 153%4

Program (Federal Agency)

Competency Development Time savings, work quality, faster response 159%4

(Veteran’s Health
Administration)

First Level Leadership Various measures – at least two per manager 105%7
Development (Auto Rental
Company)

68
 Southeast Corridor Bank

Program Title: Managing Retention

Target Group: Bank Tellers
Solution: Skills Based Pay System to Reduce Employee Turnover

RESULTS
Level 1: Level 2: Level 3: Level 4: Level 5: Intangible
Reaction Learning Application Impact ROI Benefits
Composite Positive self 95% Turnover BCR: Customer
Rating: 4.2 assessment participation reduced from 3.58 Satisfaction
out of 5 on program rate 71% to 35%
ROI: Job
understanding
86 requests Staffing level 258% Satisfaction
Positive self for training reduced by Product
assessment compared to 4% Sales
on each 46 the year
course with before Cross Selling
few
exceptions 138 review
(only two situations
failed to be 257
promoted promotions
because of compared to
performance 139 the year
in training) before

Technique to Isolate Effects of Program: Estimates from branch managers and branch
staff, adjusted for error
Technique to Convert Data to Monetary Value: External studies at similar institutions
and standard values (for staffing)
Fully-loaded Program Costs: $857,196 First year; $433,200 Second year

69
 References for Published Studies

1. In Action: Measuring Return on Investment, Volume 3. Patricia P. Phillips, Editor; Jack J.

Phillips, Series Editor. Alexandria: ASTD, 2001.
2. The Human Resources Scorecard: Measuring the Return on Investment. Jack Phillips,
Ron D. Stone, Patricia P. Phillips. Woburn: Butterworth-Heinemann, 2001.
3. In Action: Retaining Your Best Employees. Patricia P. Phillips, Editor; Jack J. Phillips,
Series Editor. Alexandria: ASTD and the Society for Human Resource Management,
2002.
4. In Action: Measuring ROI in the Public Sector. Patricia P. Phillips, Editor. Alexandria:
ASTD, 2002.
5. In Action: Coaching for Extraordinary Results. Darelyn J. Mitch, Editor; Jack J. Phillips,
Series Editor. Alexandria: ASTD, 2002.
6. In Action: Implementing Training Scorecards. Lynn Schmidt, Editor; Jack J. Phillips, Series
Editor. Alexandria: ASTD, 2003.
7. The Leadership Scorecard, Jack J. Phillips and Lynn Schmidt, Woburn: Butterworth-
Heinemann, 2004.

Additional Resources
Return on Investment in Training and Performance Improvement Programs,2nd Edition, Jack
J. Phillips. Woburn: Butterworth Heinemann, 2003.
How to Measure Training Results: A Practical Guide to Tracking the Six Key Indicators.
Jack J. Phillips and Ron D. Stone. New York: McGraw-Hill Publishing, 2002.
In Action: Measuring Intellectual Capital. Patricia P. Phillips, Editor; Jack J. Phillips, Series
Editor. Alexandria: ASTD, 2002.
Project Management Scorecard. Jack Phillips, Tim Bothell, and G. Lynne Snead. Woburn:
Butterworth Heinemann, 2002
The Bottomline on ROI. Patricia P. Phillips, Atlanta: CEP Press, 2002.
The Consultant’s Scorecard. Jack J. Phillips, New York: McGraw-Hill Publishing, 2000.
HRD Trends Worldwide: Shared Solutions to Compete in a Global Economy. Jack J.
Phillips. Woburn: Butterworth Heinemann, 1999.
In Action: Performance Analysis and Consulting. Jack J. Phillips, Editor and Series Editor,
Alexandria: ASTD, 1999.
A New Vision for Human Resources. Jac Fitz-enz and Jack J. Phillips. San Francisco: Crisp
Publications, 1998.
Accountability in Human Resource Management. Jack J. Phillips. Woburn: Butterworth
Heinemann, 1996.

70
CASE STUDY APPLICATION

71
 Retail Merchandise Company
Situation

Retail Merchandise Company (RMC) is a national chain of 420 stores, located in most
major USA markets. RMC sells small household items, gifts of all types, electronics,
and jewelry, as well as personal accessories. It does not sell clothes or major
appliances. The executives at RMC have been concerned about the slow sales growth
and were experimenting with several programs to boost sales. One of the concerns
focused on the interaction with customers. Sales associates were not actively involved
in the sales process, usually waiting for a customer to make a purchasing decision and
then proceed with processing the sale. Several store managers had analyzed the
situation to determine if more communication with the customer would boost sales. The
analysis revealed that the use of very simple techniques to probe and guide the
customer to a purchase should boost sales in each store.

The senior executives asked the training and development function to experiment with a
very simple customer interactive skills program for a small group of sales associates. A
program produced by an external supplier would be preferred to avoid the cost of
development, particularly if the program proved to be ineffective. The specific charge
from the management team was to implement the program in three stores, monitor the
results, and make recommendations. If the program increased sales and presented a
significant payoff for RMC, it should be implemented in other stores.

The sales associates are typical of the retail store employee profile. They are usually
not college graduates and most have a few months of retail store experience. Turnover
is usually quite high and formal training has not been a major part of previous sales
development efforts.

The Solution

The training and development staff conducted a very brief initial needs assessment and
identified five simple skills that would need to be covered in the program. From their
analysis, it appeared that the sales associates did not have these skills or were very
uncomfortable with the use of them. A program called “Interactive Selling Skills” was
selected – a program with significant use of skill practices. The program had two days
of training, where participants had an opportunity to practice each of the skills with a
fellow classmate, followed by three weeks of on-the-job application. Then, a final day of
training was conducted that included a discussion of problems, issues, barriers, and
concerns about using the skills. Additional practice and fine-tuning of skills was a part of
that final session. The program, an existing product from an external training supplier,
would be tried in the electronics area of three stores and sixteen people would be
trained in each store. The program was actually taught by the staff of the training
supplier for a predetermined facilitation fee.

73
Questions for discussion:

1. Which types of data does the senior management team desire?

2. Which data collection methods should be utilized for application and business impact
measurement? (Level 3 and 4)

3. Which method should be utilized to isolate the effects of the program?

4. How should the business impact data be converted to monetary values?

Notes:

74
 Data Collection Plan
Program:______________________________ Responsibility:_________________________ Date:____________

Broad Program Data Collection Data

Level Objective(s) Measures Method/Instruments Sources Timing Responsibilities
REACTION
1 AND PLANNED ACTIONS
• Positive reaction – • Rating on a • Questionnaire • Participant • End of • Facilitator
4 out of 5 composite of five program
measures (3rd day)
• Action items • Yes/No

LEARNING AND
2 CONFIDENCE
• Learn to use five simple
• Pass/Fail on skill
practice
• Observation of skill
practice by facilitator
• Facilitator • 2nd day of
program
• Facilitator

skills
APPLICATION AND
3 IMPLEMENTATION
• Initial use of five simple • Verbal Feedback • Follow-up session • Participant • 3 weeks • Facilitator
skills after 2nd
• At least 50% of participants • 5th item checked on • Follow-up • Participant day • Store Training
use all skills with every a 1 to 5 scale questionnaire • 3 months Coordinator
customer after
program
BUSINESS IMPACT
4 • Increase in sales • Weekly average
sales per sales
• Business
performance
• Company
records
• 3 months
after
• Store Training
Coordinator
associate monitoring program

ROI Comments:_______________________________________________________________________________
5 • 50% ________________________________________ ________________________________________________

75
 ROI Analysis Plan

Program: __________________________ Responsibility: ___________________ Date: _____________

Methods of
Isolating the
Effects of the Methods of Intangible Communication Other
Data Items Program Converting Data Cost Categories Benefits Targets Influences/Issues

• Weekly • Control group • Direct • Facilitation Fees • Customer • Program • Must have job
Sales Per analysis conversion • Program satisfaction participants coverage during
Associate • Participant using profit Materials • Employee • Electronics training
estimate contribution • Meals/ satisfaction Dept. • No
Refreshments Managers – communication
• Facilities Target Stores with control
• Participant • Store group
Salaries/Benefits Managers – • Seasonal
• Cost of Target Stores fluctuations
Coordination/ • Senior Store should be
Evaluation Executives avoided
District,
Region,
Headquarters
• Training
Staff:
Instructors,
Coordinators,
Designers,
and
Managers

76
 Program Profile
Title: Interactive Selling Skills
Target Group: Sales Associates in Electronics
• Vendor produced and delivered
• 3-days – (2 days plus 1 day)
• Significant use of skill practices
• Three groups trained (48 participants from three stores)

ROI Analysis Profile

Post Program Data Collection
• (4) Performance Monitoring 3 months
• (3) Questionnaire 3 months
• (3) Program follow-up session 3 weeks (last session)

Isolating the Effects of Training

• Control group arrangement – selected on:
o Store size
o Store location
o Customer traffic levels
o Previous store performance
• Participant’s estimate (for back-up)

Converting Data to Monetary Values

• Profit contribution of increased output

Level 1 – Selected Data

Success with objectives 4.3
Relevance of material 4.4
Usefulness of program 4.5
Exercises / skill practices 3.0
Overall instructor rating 4.1

77
 Level 2 – Selected Data

All participants demonstrated that they

could use the skills successfully.

Level 3 – Selected Data

(Two questions out of 20)
I used the Neither
skills taught Strongly agree nor Strongly
in the agree Agree disagree Disagree disagree
program 78% 22% 0% 0% 0%
Frequency Several At least
of use of With each Every third times each At least once
skills customer customer day once daily weekly
52% 26% 18% 4% 0%

Level 4 – Average Weekly Sales

Post Training Data
Weeks After Training Trained Groups Control Groups
1 $ 9,723 $ 9,698
2 9,978 9,720
3 10,424 9,812
13 $13,690 $11,572
14 11,491 9,683
15 11,044 10,092

Average for weeks 13, 14, and 15 $ 12,075 $ 10,449

Notes:

78
 Annualized Program Benefits

(46 participants were still in the job after three months)

Average weekly sales per employee (trained groups) $12,075

Average weekly sales per employee (untrained groups) 10,449
Increase 1,626
Profit contribution (2% of store sales) 32.50
Total weekly improvement (32.50 x 46) 1,495
Total annual benefits (1,495 x 48 weeks) $71,760

Cost Summary

(48 participants in three courses)

Facilitation fees (3 courses @ $3,750) $ 11,250

Program materials (48 @ $35/participant) 1,680
Meals/refreshments (3 days @ $28/participant) 4,032
Facilities (9 days @$120) 1,080
Participant salaries plus benefits (35% factor) 12,442
Coordination / Evaluation 2,500
Total Cost $ 32,984

Level 5 Data

BCR =

ROI (%) = x 100

Notes:

79
 ROI Example
Tabulating
program costs
($32,984)

Collecting Isolating the Converting Calculating the

post-program effects of the data to return on
data program monetary value investment
($71,760) (118%)
• Follow-up • Control group • Standard
session • Participants’ values
• Questionnaire estimates
• Performance
Identifying
monitoring intangible
benefits

The ROI Process Takes a Balanced View of Evaluation

by Measuring and Reporting:
• Reaction to program
• Application on the job
• The financial results
• Nature and source of problems and
opportunities
• Learning and attitudes
• Impact in work unit
• Intangible benefits
• Impact on the customer

Notes:

80

Increase your credibility by proving your
contribution to financial results
Fully master all the skills you need to implement a
comprehensive, on-going ROI initiative in your
organization in this five-day workshop. You'll gain the
skills to become certified in the ROI Methodology™ and
you'll learn how to sustain the measurement and
evaluation process on a continued basis. You'll also
receive on-going, personalized coaching once the
workshop has ended as you conduct an impact study in
your own organization. Once you've demonstrated
competency in applying the ROI Methodology™, you will
be certified. No other workshop gives you access to the
same level of expertise as ROI Certification.
Who Should Attend
Anyone who needs the skills to measure the impact of
learning and performance improvement programs.
Individuals who need to demonstrate the ROI and
business impact of other initiatives such as human
resources, quality, process improvement, organizational
change, and technology implementation will also benefit.
Certification Benefits
• Demonstrated competency in conducting an impact
study
• A Certificate of Accomplishment (upon successful
completion of your own impact study)
• Permission to conduct one-day workshops on the ROI
Methodology™ in your own organization
• Sought-after skills that will make you more credible
and valuable to your organization's executives
ROI Institute, Inc.
As an additional benefit, you will become a member of
The ROI Institute, the leading source for research and
case studies on the application of ROI around the world.
As a member of the ROI Institute, you will be an
authorized practitioner of the ROI Methodology™ and
entitled to give one day workshops. You will also have
access to the Members Only area of our website –
www.roiinstitute.net, select the Members Only link.
Contact Information and Registration - Please contact ROI Institute, Inc. at 205-678-8101 or info@roiinstitute.net
© 2006 ROI Institute, Inc.
81
The ROI Methodology™
Certification Workshop
What You Will Learn
During the workshop, you will:
• Apply the ROI Methodology™ to a real program from
your own organization
• Get detailed, first hand experience with every step
• Learn how to calculate ROI
• Learn how to measure the contribution of training and
performance improvement programs
• Learn how to enhance program results and improve
ROI
• Explore the most common reasons that ROI initiatives
fail and learn how to avoid them
• Learn how to translate both tangible and intangible
benefits into monetary values
Gain the skills to be fully certified, including
how to:
• Determine the timing of and select data collection
instruments
• Select appropriate strategies for isolating the effects of
programs
• Calculate or estimate the value of an improvement
• Apply simple statistical tools to data analysis
• Analyze data and calculate the actual return on
investment
Facilitation
Facilitated by Jack Phillips, Ph.D., and Patti Phillips,
Ph.D.,, the ROI Methodology™ certification workshop
provides a practical measurement experience. All three
facilitators have authored several publications on the
subject of accountability and ROI and are renowned
experts in the field of measurement and evaluation.
Certification Workshop Dates and Sites
We offer Certification Workshops to the public throughout
the year at sites worldwide. We also provide internal
Certification Workshops customized for your business or
organization.
Please visit our website at www.roiinstitute.net to learn
more and see the upcoming dates for public workshops in
the United States and around the world.
Phone: 205-678-8101 ∗ Fax: 205-678-8102
Email: info@roiinstitute.net
www.roiinstitute.net
 The ROI Fact Sheet

Origin/Development
• The ROI Methodology™ was developed by Dr. Jack J. Phillips in the 1970s, refined through
application and use in the 1980s, and implemented globally during the 1990s.
• First impact study – 1973, Measuring the ROI in a Cooperative Education Program, for Lockheed-
Martin
• First public presentation on the methodology – 1978, ASTD Annual Conference
• First book published to include methodology – 1983, Handbook of Training Evaluation and
Measurement Methods, Gulf Publishing (this was the first USA book on training evaluation)
• First one-day public workshop –1991, Birmingham, Alabama
• First two-day public workshop –1992, Johannesburg, South Africa
• First case study book published – 1994, Measuring Return on Investment, ASTD
• First international partnership established – 1994, Indonesia
• First public certification workshop – 1995, Nashville, Tennessee
• ROI Network organized - 1996
• First ROI Network Conference –1997, New Orleans, Louisiana
• First international ROI Network conference – 2002, Toronto, Canada
• First ROI in Government Conference – 2003, Gulfport, Mississippi, Co-sponsored by the University of
Southern Mississippi
• First ROI software release – 2003, KnowledgeAdvisors
• On-line ROI certification launched – 2006, University Alliance
• ROI Certification offered as part of Masters and Ph.D. degree – Capella University, 2006.

Use
• Over 3,000 organizations are using the ROI methodology, through planned implementation.
• 2,000 organizations have formally implemented the methodology through ROI Certification™
conducted by the ROI Institute.
• approximately 5,000 impact studies are conducted annually in learning and development and human
resources.
• At least 200 public sector governmental units are using the methodology.
• ROI implementation was first pursued in manufacturing, then moved to service, healthcare, non-
profits, governments, and is now in educational systems

Applications
Typical applications include:

• Apprenticeship • Gainsharing • Safety & Health Programs

• Career Management
• Competency Systems
• Diversity
• E-Learning
• Coaching
• Information Assurance
• Meetings and Events
• Management Development
• Leadership Development
• Organization Development
• Orientation
• Recruiting
• Self-Directed Teams
• Skill-Based/Knowledge-
Based Compensation
• Technology Implementation
• Total Quality Management
• Wellness/Fitness Initiatives

Articles and Publicity

• Over 60 articles have been published on the ROI methodology in major publications in 20 countries.
• The ROI methodology has been a cover story on at least 15 publications, magazines, and journals.
• At least 50 interviews in major global business and professional publications
• Over 25 radio and TV interviews in different countries
© 2006, ROI Institute, Inc.

Phone: 205-678-8101 ∗ Fax: 205-678-8102

Email: info@roiinstitute.net

82
 The ROI Fact Sheet

Books
• Sixteen books have been published on the ROI methodology and its application (www.roiinstitute.net)
• Primary reference – Return on Investment in Training and Performance Improvement Projects, 2nd
Edition, Jack J. Phillips, Butterworth-Heinemann, Woburn, MA, 2003 (originally published in 1997)
• Award winning book – Bottomline on ROI, Patti P. Phillips, CEP Press, Atlanta, GA, 2002 (received
ISPI award)

Case Studies
• Over 100 case studies published in books, journals, and industry publications
• Four-volume set published by ASTD in 1994, 1997, 2001, and 2005
• First public sector case book – 2002, published jointly by the International Personnel and
Management Association and the American Society for Training and Development
• First International case book – 2005, Ireland published by Skillnets
• International case studies under development in 12 countries

Workshops (One-Day, Two-Day, and Three- Day)

• Approximately 200 one-day workshops conducted with over 8,000 participants
• Approximately 500 two-day workshops conducted with more than 15,000 specialists and managers
attending (offered in almost every major international city)
• Routine schedules of one-day, two-day, and three-day workshops offered in the USA by ASTD
(www.astd.org) and through partners around the world

ROI Certification™
• Five-day workshop plus two work products lead to certification for ROI implementation
• Over 3,000 professionals have attended certification, representing over 2,000 organizations in at
least 50 countries
• Certifications offered routinely about 25 times per year both internally and publicly by the ROI
Institute (www.roiinstitute.net)
• On-line certification begins every month-six months duration (www.roiinstituteonline.com)

Global Implementation
• First implementation of the ROI methodology outside the USA – 1992, South Africa
• First certification in non-English language – 1995, Italy
• Implementation is accomplished through partners in various countries
• Implementation is currently occurring in 44 countries, with additional implementations planned in
other countries
• Books published in 28 languages
ƒ Twelve international case study books in development or in the planning stages

Phone: 205-678-8101 ∗ Fax: 205-678-8102

Email: info@roiinstitute.net

83
 Jack J. Phillips, Ph.D.

As a world-renowned expert on accountability, measurement and evaluation, Dr. Phillips provides

consulting services for Fortune 500 companies and major global organizations. The author or
editor of more than 50 books, Phillips provides workshops and conference presentations
throughout the world.

His expertise in measurement and evaluation is based on more than twenty-seven years of
corporate experience in five industries (aerospace, textiles, metals, construction materials, and
banking). Phillips has served as training and development manager at two Fortune 500 firms,
senior HR officer at two firms, president of a regional bank, and management professor at a major
state university.

His background led Phillips to develop the ROI Methodology⎯a revolutionary process that
provides bottom-line figures and accountability for all types of learning, performance improvement,
human resources, technology, and public policy programs.

Phillips regularly consults with clients in manufacturing, service, and government organizations in
44 countries in North and South America, Europe, Africa, Australia, and Asia

Books most recently authored by Phillips include Investing in Your Company’s Human Capital:
Strategies to Avoid Spending Too Much or Too Little, Amacom 2005; Proving the Value of HR:
How and Why to Measure ROI, SHRM 2005; The Leadership Scorecard, Elsevier Butterworth-
Heinemann 2004; The Human Resources Scorecard: Measuring the Return on Investment,
Elsevier Butterworth-Heinemann 2001; Building a Successful Consulting Practice, ASTD 2002;
The Consultant’s Scorecard, McGraw-Hill 2000; Managing Employee Retention, Elsevier
Butterworth-Heinemann, 2003; Return on Investment in Training and Performance Improvement
Projects, 2nd Edition Elsevier Butterworth-Heinemann 2003; The Project Management Scorecard,
Elsevier Butterworth-Heinemann 2002; How to Measure Training Results, McGraw-Hill 2002; and
Performance Analysis and Consulting, ASTD 2000. Phillips served as series editor for ASTD’s In
Action casebook series, one of ASTD’s more ambitious publishing projects with 30 titles. Now, he
serves as series editor for Elsevier Butterworth-Heinemann’s Improving Human Performance
series and Pfeiffer’s new series on Measurement and Evaluation.

Phillips has received several awards for his books and his work. The Society for Human Resource
Management gave him its highest creative award for an ROI study and an award for one of his
books. The American Society for Training and Development gave him its highest award,
Distinguished Contribution to Workplace Learning and Development. Meeting News named Phillips
one of the 25 most influential people in the Meetings and Events industry, based on his work on
ROI for the industry.

Phillips has undergraduate degrees in electrical engineering, physics, and mathematics; a master’s
degree in decision sciences from Georgia State University; and a Ph.D. in human resource
management from the University of Alabama.

Jack Phillips has served on several boards of private businesses – including two NASDAQ
companies – and several non-profits and associations, including the American Society for Training
and Development. He is Chairman, ROI Institute, Inc. and can be reached at (205) 678-8101, or by
e-mail at jack@roiinstitute.net.

84