Kirkpatrick's Four-Level Training Evaluation Model

Analyzing Training Effectiveness

Evaluate training effectively. iStockphoto/Jeryco

If you deliver training for your team or your organization, then you probably know how important it is to measure its effectiveness. After all, you don't want to spend time or money on training that doesn't provide a good return. This is where Kirkpatrick's Four-Level Training Evaluation Model can help you objectively analyze the effectiveness and impact of your training, so that you can improve it in the future. In this article, we'll look at each of the Kirkpatrick four levels, and we'll examine how you can apply the model to evaluate training. We'll also look at some of the situations where the model may not be useful.

The Four Levels

Donald Kirkpatrick, Professor Emeritus at the University of Wisconsin and past president of the American Society for Training and Development (ASTD), first published his Four-Level Training Evaluation Model in 1959, in the US Training and Development Journal. The model was then updated in 1975, and again in 1994, when he published his best-known work, "Evaluating Training Programs." The four levels are: 1. Reaction. 2. 3. 4. Learning. Behavior. Results.

Let's look at each level in greater detail.

Level 1: Reaction
This level measures how your trainees (the people being trained), reacted to the training. Obviously, you want them to feel that the training was a valuable experience, and you want them to feel good about the instructor, the topic, the material, its presentation, and the venue. It's important to measure reaction, because it helps you understand how well the training was received by your audience. It also helps you improve the training for future trainees, including identifying important areas or topics that are missing from the training.

Level 2: Learning
At level 2, you measure what your trainees have learned. How much has their knowledge increased as a result of the training? When you planned the training session, you hopefully started with a list of specific learning objectives: these should be the starting point for your measurement. Keep in mind that you can measure learning in different ways depending on these objectives, and depending on whether you're interested in changes to knowledge, skills, or attitude. It's important to measure this, because knowing what your trainees are learning and what they aren't will help you improve future training.

Level 3: Behavior
At this level, you evaluate how far your trainees have changed their behavior, based on the training they received. Specifically, this looks at how trainees apply the information.

It's important to realize that behavior can only change if conditions are favorable. For instance, imagine you've skipped measurement at the first two Kirkpatrick levels and, when looking at your group's behavior, you determine that no behavior change has taken place. Therefore, you assume that your trainees haven't learned anything and that the training was ineffective. However, just because behavior hasn't changed, it doesn't mean that trainees haven't learned anything. Perhaps their boss won't let them apply new knowledge. Or, maybe they've learned everything you taught, but they have no desire to apply the knowledge themselves.

Level 4: Results
At this level, you analyze the final results of your training. This includes outcomes that you or your organization have determined to be good for business, good for the employees, or good for the bottom line. Tip: Make sure that you plan your training effectively. Use our articles on Training Needs Assessment, Gagne's Nine Levels of Learning and 4MAT to help you do this.

How to Apply the Model

Level 1: Reaction
Start by identifying how you'll measure reaction. Consider addressing these questions: Did the trainees feel that the training was worth their time?

Did they think that it was successful? What were the biggest strengths of the training, and the biggest weaknesses? Did they like the venue and presentation style?

Did the training session accommodate their personal learning styles? Next, identify how you want to measure these reactions. To do this you'll typically use employee satisfaction surveys or questionnaires; however you can also watch trainees' body language during the training, and get verbal feedback by asking trainees directly about their experience. Once you've gathered this information, look at it carefully. Then, think about what changes you could make, based on your trainees' feedback and suggestions.

Level 2: Learning
To measure learning, start by identifying what you want to evaluate. (These things could be changes in knowledge, skills, or attitudes.) It's often helpful to measure these areas both before and after training. So, before training commences, test your trainees to determine their knowledge, skill levels, and attitudes. Once training is finished, test your trainees a second time to measure what they have learned, or measure learning with interviews or verbal assessments.

Level 3: Behavior
It can be challenging to measure behavior effectively. This is a longer-term activity that should take place weeks or months after the initial training. Consider these questions: Did the trainees put any of their learning to use?

Are trainees able to teach their new knowledge, skills, or attitudes to other people? Are trainees aware that they've changed their behavior?

One of the best ways to measure behavior is to conduct observations and interviews over time. Also, keep in mind that behavior will only change if conditions are favorable. For instance, effective learning could have taken place in the training session. But, if the overall organizational culture isn't set up for any behavior changes, the trainees might not be able to apply what they've learned. Alternatively, trainees might not receive support, recognition, or reward for their behavior change from their boss. So, over time, they disregard the skills or knowledge that they have learned, and go back to their old behaviors.

Level 4: Results
Of all the levels, measuring the final results of the training is likely to be the most costly and time consuming. The biggest challenges are identifying which outcomes, benefits, or final results are most closely linked to the training, and coming up with an effective way to measure these outcomes over the long term. Here are some outcomes to consider, depending on the objectives of your training: Increased employee retention.

Increased production.

Higher morale. Reduced waste. Increased sales. Higher quality ratings. Increased customer satisfaction. Fewer staff complaints.

Considerations
Although Kirkpatrick's Four-Level Training Evaluation Model is popular and widely used, there are a number of considerations that need to be taken into account when using the model. One issue is that it can be time-consuming and expensive to use levels 3 or 4 of the model, so it's not practical for all organizations and situations. This is especially the case for organizations that don't have a dedicated training or human resource department, or for one-off training sessions or programs. In a similar way, it can be expensive and resource intensive to "wire up an organization" to collect data with the sole purpose of evaluating training at levels 3 and 4. (Whether or not this is practical depends on the systems already in place within the organization.) The model also assumes that each level's importance is greater than the last level, and that all levels are linked. For instance, it implies that Reaction is less important, ultimately, than Results, and that reactions must be positive for learning to take place. In practice, this may not be the case. Most importantly, organizations change in many ways, and behaviors and results change depending on these, as well as on training. For example, measurable improvements in areas like retention and productivity could result from the arrival of a new boss or from a new computer system, rather than from training. Kirkpatrick's model is great for trying to evaluate training in a "scientific" way, however, so many variables can be changing in fast-changing organizations that analysis at level 4 can be limited in usefulness.

Key Points
The Kirkpatrick Four-Level Training Evaluation Model helps trainers to measure the effectiveness of their training in an objective way. The model was originally created by Donald Kirkpatrick in 1959, and has since gone through several updates and revisions. The Four-Levels are as follows: 1. Reaction. 2. 3. 4. Learning. Behavior. Results.

By going through and analyzing each of these four levels, you can gain a thorough understanding of how effective your training was, and how you can improve in the future. Bear in mind that the model isn't practical in all situations, and that measuring the effectiveness of training with the model can be time-consuming and use a lot of resources.

kirkpatrick's learning and training evaluation theory

Donald L Kirkpatrick's training evaluation model the four levels of learning evaluation also below - HRD performance evaluation guide

Donald L Kirkpatrick, Professor Emeritus, University Of Wisconsin (where he achieved his BBA, MBA and PhD), first published his ideas in 1959, in a series of articles in the Journal of American Society of Training Directors. The articles were subsequently included in Kirkpatrick's book Evaluating Training Programs (originally published in 1994; now in its 3rd edition - Berrett-Koehler Publishers). Donald Kirkpatrick was president of the American Society for Training and Development (ASTD) in 1975. Kirkpatrick has written several other significant books about training and evaluation, more recently with his similarly inclined son James, and has consulted with some of the world's largest corporations. Donald Kirkpatrick's 1994 book Evaluating Training Programs defined his originally published ideas of 1959, thereby further increasing awareness of them, so that his theory has now become arguably the most widely used and popular model for the evaluation of training and learning. Kirkpatrick's fourlevel model is now considered an industry standard across the HR and training communities. More recently Don Kirkpatrick formed his own company, Kirkpatrick Partners, whose website provides information about their services and methods, etc.

kirkpatrick's four levels of evaluation model

The four levels of Kirkpatrick's evaluation model essentially measure:

reaction of student - what they thought and felt about the training learning - the resulting increase in knowledge or capability behaviour - extent of behaviour and capability improvement and implementation/application results - the effects on the business or environment resulting from the trainee's performance

All these measures are recommended for full and meaningful evaluation of learning in organizations, although their application broadly increases in complexity, and usually cost, through the levels from level 1-4.

Quick Training Evaluation and Feedback Form, based on Kirkpatrick's Learning Evaluation Model (Excel file) kirkpatrick's four levels of training evaluation
This grid illustrates the basic Kirkpatrick structure at a glance. The second grid, beneath this one, is the same thing with more detail.
level evaluation type (what is measured) 1 Reaction evaluation description and characteristics examples of evaluation tools and methods relevance and practicability

Reaction evaluation ishow the delegates feltabout the training or learning experience.

'Happy sheets', feedback forms. Verbal reaction, posttraining surveys or questionnaires.

Quick and very easy to obtain. Not expensive to gather or to analyse.

Learning

Learning evaluation is the measurement of theincrease in knowledge - before and after.

Typically assessments or Relatively simple to set up; tests before and after clear-cut for quantifiable the training. skills. Interview or observation can also be used. Less easy for complex learning.

Behaviour

Behaviour evaluation is the extent of applied learning back on the job - implementation.

Observation and interview over time are required to assess change, relevance of change, and sustainability of change.

Measurement of behaviour change typically requires cooperation and skill of linemanagers.

Results

Results evaluation is theeffect on the business or environment by the trainee.

Measures are already in place via normal management systems and reporting - the challenge is to relate to the trainee.

Individually not difficult; unlike whole organisation. Process must attribute clear accountabilities.

kirkpatrick's four levels of training evaluation in detail

This grid illustrates the Kirkpatrick's structure detail, and particularly the modern-day interpretation of the Kirkpatrick learning evaluation model, usage, implications, and examples of tools and methods. This diagram is the same format as the one above but with more detail and explanation:
evaluation level and type 1. Reaction evaluation description and characteristics examples of evaluation tools and methods relevance and practicability

Reaction evaluation is how the delegates felt, and theirpersonal reactions to the training or learning experience, for example: Did the trainees like and enjoy the training? Did they consider the training relevant? Was it a good use of their time? Did they like the venue, the style, timing, domestics, etc? Level of participation. Ease and comfort of experience. Level of effort required to make the most of the learning. Perceived practicability and potential for applying the learning.

Typically 'happy sheets'. Feedback forms based on subjective personal reaction to the training experience. Verbal reaction which can be noted and analysed. Post-training surveys or questionnaires. Online evaluation or grading by delegates. Subsequent verbal or written reports given by delegates to managers back at their jobs.

Can be done immediately the training ends. Very easy to obtain reaction feedback Feedback is not expensive to gather or to analyse for groups. Important to know that people were not upset or disappointed. Important that people give a positive impression when relating their experience to others who might be deciding whether to experience same.

2. Learning

Learning evaluation is the measurement of the increase in knowledge or intellectual capability from before to after the learning

Typically assessments or tests before and after the training. Interview or observation can

Relatively simple to set up, but more investment and thought required than reaction evaluation.

experience: Did the trainees learn what what intended to be taught? Did the trainee experience what was intended for them to experience? What is the extent of advancement or change in the trainees after the training, in the direction or area that was intended?

be used before and after although this is timeconsuming and can be inconsistent. Methods of assessment need to be closely related to the aims of the learning. Measurement and analysis is possible and easy on a group scale. Reliable, clear scoring and measurements need to be established, so as to limit the risk of inconsistent assessment. Hard-copy, electronic, online or interview style assessments are all possible.

Highly relevant and clearcut for certain training such as quantifiable or technical skills. Less easy for more complex learning such as attitudinal development, which is famously difficult to assess. Cost escalates if systems are poorly designed, which increases work required to measure and analyse.

3. Behaviour

Behaviour evaluation is the extent to which the traineesapplied the learning andchanged their behaviour, and this can be immediately and several months after the training, depending on the situation: Did the trainees put their learning into effect when back on the job? Were the relevant skills and knowledge used

Observation and interview over time are required to assess change, relevance of change, and sustainability of change. Arbitrary snapshot assessments are not reliable because people change in different ways at different times. Assessments need to be subtle and ongoing, and then transferred to a suitable analysis tool.

Measurement of behaviour change is less easy to quantify and interpret than reaction and learning evaluation. Simple quick response systems unlikely to be adequate. Cooperation and skill of observers, typically linemanagers, are important factors, and difficult to control. Management and analysis of ongoing subtle assessments are difficult, and virtually impossible without a well-designed system from the beginning. Evaluation of implementation and application is an extremely important assessment - there is little point in a good

Was there noticeable and measurable change in the Assessments need to be activity and performance of the designed to reduce subjective trainees when back in their judgement of the observer or roles? interviewer, which is a variable factor that can affect Was the change in behaviour reliability and consistency of and new level of knowledge measurements. sustained? The opinion of the trainee, Would the trainee be able to which is a relevant indicator, transfer their learning to is also subjective and another person? unreliable, and so needs to be measured in a consistent Is the trainee aware of their

change in behaviour, knowledge, skill level?

defined way. 360-degree feedback is useful method and need not be used before training, because respondents can make a judgement as to change after training, and this can be analysed for groups of respondents and trainees. Assessments can be designed around relevant performance scenarios, and specific key performance indicators or criteria. Online and electronic assessments are more difficult to incorporate assessments tend to be more successful when integrated within existing management and coaching protocols. Self-assessment can be useful, using carefully designed criteria and measurements.

reaction and good increase in capability if nothing changes back in the job, therefore evaluation in this area is vital, albeit challenging. Behaviour change evaluation is possible given good support and involvement from line managers or trainees, so it is helpful to involve them from the start, and to identify benefits for them, which links to the level 4 evaluation below.

4. Results

Results evaluation is theeffect on the business or environment resulting from the improved performance of the trainee - it is the acid test. Measures would typically be business or organisational key performance indicators, such as: Volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organisational performance, for instance; numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations,

It is possible that many of these measures are already in place via normal management systems and reporting. The challenge is to identify which and how relate to to the trainee's input and influence. Therefore it is important to identify and agree accountability and relevance with the trainee at the start of the training, so they understand what is to be measured. This process overlays normal good management practice it simply needs linking to the

Individually, results evaluation is not particularly difficult; across an entire organisation it becomes very much more challenging, not least because of the reliance on line-management, and the frequency and scale of changing structures, responsibilities and roles, which complicates the process of attributing clear accountability. Also, external factors greatly affect organisational and business performance, which cloud the true

growth, retention, etc.

training input. Failure to link to training input type and timing will greatly reduce the ease by which results can be attributed to the training. For senior people particularly, annual appraisals and ongoing agreement of key business objectives are integral to measuring business results derived from training.

cause of good or poor results.

Since Kirkpatrick established his original model, other theorists (for example Jack Phillips), and indeed Kirkpatrick himself, have referred to a possible fifth level, namely ROI (Return On Investment). In my view ROI can easily be included in Kirkpatrick's original fourth level 'Results'. The inclusion and relevance of a fifth level is therefore arguably only relevant if the assessment of Return On Investment might otherwise be ignored or forgotten when referring simply to the 'Results' level. Learning evaluation is a widely researched area. This is understandable since the subject is fundamental to the existence and performance of education around the world, not least universities, which of course contain most of the researchers and writers. While Kirkpatrick's model is not the only one of its type, for most industrial and commercial applications it suffices; indeed most organisations would be absolutely thrilled if their training and learning evaluation, and thereby their ongoing people-development, were planned and managed according to Kirkpatrick's model.