Putting it all Together

In this chapter, you have learned many factors that can affect the success of a training program. Before discussing how the success of a training program can be evaluated, lets recap what you have learned. As shown in Exhibit 8.07, the first issue to consider is whether training is the proper solution to a problem. That is, if employees already possess the necesarry skills and knowledge but arent performing well, the problem is probably one of motivation, communication, or work design rather than a lack of training. If training is to be the desired intervention, several factors will affect its success: Employees must have the skills and abilities to complete the training successfully. For example, if an employee is not bright enough to learn a computer program or doesnt have the dexterity to perform intricate microelectronics assembly, no amount of training will improve his performance. There should be minimal outside factors (e.g., work or family problems) that might distract the employee and keep him from concentrating on the training program. Employees must be motivated to learn. That is, they must perceive that training is needed, that the training program meets their expectations, that they have the ability to complete the training (self-efficacy), and that there will be some reward (e.g., pay, career advancement) for performing well. The training method (e.g., programmed instruction, behavioral modeling, lecture) must be a good match for the employees learning style, ability, and personality. The training method must be a good match for the type of material being learned (e.g., knowledge vs. A physical skill). The training program must allow for goal setting, positive feedback, distributed practice, overlearning, and the chance to practice and apply the material learned (transfer of training). There must be an opportunity and encouragement to use the newly acquired knowledge at work.

Evaluation of Training Results

As discussed in Chapter 1, one important characteristic of industrial psychology is its reliance on research. Evaluating training results is a good example of this reliance. Because training programs can be costly in both time and money, it is essential that they be evaluated to determine if they can be improved and should continue to be offered, and whether they

significantly increase performance or effect positive changes in behavior (Kirkpatrick, 2000). A survey training directors, however, found that 90% claimed that, even though tehy believed the evaluation of training to be important, they did not require them (Bell & Kerr, 1987).

Research Designs for Evaluation

There are many ways to evaluate the effectiveness of a training program, and two factors differentiate the various methods. The first involves practicality, and the second is concerned with experimental rigor. Although scientifically rigorous research designs are preferred, their use is not always possible. Yet a practical research design without scientific rigor yields little confidence in research findings. The most simple and practical of research designs implements a training program and then determines whether significant change is seen in performance or job knowledge. To use this method, performance or job knowledge must be measured twice. The first measurement, a pretest, is taken before the implementation of training. The second mesurement, a posttest, is taken after the training program is complete. A diagram of this simple pretest-posttest design is as follows: Pretest Training Posttest Although this method is fairly simple, its findings are difficult to interpret because there is no control group against which the results can be compared. That is, suppose a significant difference in performance is seen between the pretest and the posttest. If a training program has occured between the two test, it would be tempting to credit the training for the increase. The increase, however, may have resulted from other factor such as changes in machinery, changes ini motivation caused by nontraining factors, or changes in managerial style or philosophy. Likewise, suppose no significant increase in performance in observed between pretest and posttest. The natural conclusion might be that the training program did not work. Without a control group, that interpretation is not necessarily correct. The same changes noted above for an increase may have caused a decrease in performance in this second case. Thus, it is possible that the training program actually did increase performance but that other factors reduced it, which resulted in no net gain in performance from training. To overcome these problems, a control group should be used (Kearns, 2001). For training purposes, a control group consists of employees who will be tested and treated in the same manner as the experimental group except that they will not receive training. The control group will be subject to the same policy, machinery, and economic conditions as the

employees in the experimental group who receive training. The diagram for a pretest-posttest control group design looks like this: Experimental group Control group : : Pretest Pretest Training Posttest Posttest

The big advantage this second design has it that it allows a researcher to look at the training effect controlling for outside factors. For example, after going through a training program, employees at R.R. Donnelley & Sons increased their annual commissions by $22,000, and the company was obviously pleased but was worried that the increased performance could have been due to something other than training. So it compared the results to a control group of employees who had not received training. The commissions of the control employees increased by $7,000 over the same period. Thus, the net effect of the training was still sizable- $15,000 per employee- but not as high as the $22,000 originally thought. The control group allowed the company to control for such factors as increased sales agent experience and new company promotions (Montebello & Haga, 1994). Even though this design is an improvement on the first, it too has its drawbacks. First, except for training manipulation, it is almost impossible to treat a control group the same as the experimental group. Control groups often consist of workers at other plants or on other shifts at the same plant. Such groups are used because there often is no alternative. But the fact that they are in different environments reduces confidence in the research findings. Even if employees in the same plant on the same shift can be randomly split into control and experimental groups, problems will still exist. The most glaring of these involves the possibility that because the two groups are close to one another, the training effect for the experimental group will spill over to the control group. Employees in the control group also may resent not being chosen for training. This resentment alone may lead to a decrease in performance by employees in the control group. Finally, it is possible that the untrained employees will pressure the newly trained employees will pressure the newly trained employees to revert to the old way of doing things (Spitzer, 1986). With both of the above designs, the pretest itself presents a problem. That is, the mere taking of a test may itself lead to increase in performance. Because of this, a rather complicated method called the Solomon four-groups design can be used (Campbell & Stanley, 1963). With this design, one group will undergo trining but will not take the pretest, a second group will undergo training but will take the pretest, a third group will not undergo training but will take the pretest, and a fourth group will neither undergo training nor take the pretest. The diagram for this design is as follow:

Group 1 Group 2 Group 3 Group 4 Pretest Pretest

Training Training

Posttest Posttest Posttest Posttest

This design allows a researcher not only to control for outside effects but also to control for any pretest effect. This is the most scientifically rigorous of the research designs used to evaluate training, but even it has a serious drawback: It is often not practical. That is, four groups of employees must be used, two of which do not receive training. Thus, to use this design at one organization or plant, ideally a relatively large number of employees must be available and kept from discussing the training with one another.

Evaluation Criteria
In the previous section, we discussed research designs for evaluating training. In each design, a pretest and posttest were included. This section will discuss the types of criteria that can be used for these pre- and posttests. There are five levels at which training effectiveness can be measured: content validity, employee reactions, employee learning, application of training, and bottom-line measures.

Content Validity At times, the only way that training can be evaluated is by comparing training content with the knowledge, skills, and abilities required to perform a job. In other words, the content validity of the training can be examined. For example, if a job analysis indicates that knowledge of electronic circuitry is necessary to perform a job, then a seminar that is designed to teach this knowledge would have content validity. Although content analysis may ensure that a training program is job related, it still does not indicate whether a particular training method is effective. But if a training program is content valid and is conducted by a professional trainer who can document previous success with the method in other organizations, it may be s safe assumption that the training program will be successful. Keep in mind, however, that making such an assumption is acceptable only when actually evaluating the effect of training is not possible because there are too few employees.

Employee Reactions

The most commonly used method to evaluate training is measuring employee reactions to the training (Tyler, 2002). Thus, when directly measuring training effects is not possible, trainee reactions can be used. Employee reactions are important because employee will not have confidence in the training and will not be motivated to use it if they do not like the training process. However, even though positive employee reactions are necessary for training to be successful, positive employee reactions do not mean that training will lead to changes in knowledge or performance (Pfau & Kay, 2002b). In fact, trainee reactions constitute the lowest level of training evaluation (Birnbrauer, 1987) and can often be misleading. For example, most seminars conducted by outside consultants are informative and well presented, so employee reactions are almost always positive, even though the training may not actually affect knowledge or future performance. For example, as shown in Exhibit 8.08, in a meta-analysis by Alliger, Exhibit 8.08 Tannenbaum, Bennet, Traver, and Shotland (1997), employee reactions had only a small correlation with lerning and application of training.

Employee Learning Instead of using employee reactions as the criterion in evaluating training performance, actual employee learning can usually be measured (Tyler, 2002). That is, if a training program is design to increase employee knowledge of communication techniques, then creating a test to determine whether an employee actually learned is possible. This test will be administered before training and then again after the training has been completed. The measurements that will be used for the pretest and posttest, as with selection tests, must be both reliable and valid. Thus, if the purpose of a training program is to increase job knowledge, an appropriate job knowledge test must be constructed or purchased. A trainer can spend a great deal of time creating a training program and evaluating its effectiveness, but the whole training effort will be wasted if the measure used to evaluate effectiveness is no good. The restaurant industry provides many examples of evaluating training effectiveness through employee learning. At Claim Jumpers Restaurant, servers must pass a 100-item menu test before they are released from training. At Rock Bottom Restaurants, learning is measured by requiring new employees to obtain written statements from their peers verifying that they have mastered their new duties; 20% do not perform well enough to get the required verificaton. To make the testing process more fun, Bugaboo Creek Restaurants hold scavenger hunts to ensure that trainees know where everything is located (Zuber, 1996).

At times, reliable and valid measures of training effectiveness are difficult to obtain. Perhaps a good example of this is seen with the human relations seminars that are common to traininig programs (Buzzotta, 1986). Typlically, an outside consultant conducts a seminar on a toppic such as better communication skills or calming irate customers. A seminar may run from 2 hours to 2 days in length. Once completed, however, it is important to measure the effectiveness of the seminar training.

Application of Training Another criterion for evaluating the effectiveness of training is the degree application of training, or the extent to which employees actually can use the learned material (Geber, 1995). Learning and memorizing new material is one thing, and applying it is another. For example, if employees learn how to deal with angry customers, their ability to apply this material can be measured by observing how they treat an angry customer while they are actually working.

Bottom-Line Measures The final criterion that can be used to evaluate a training programs effectiveness is a bottom-line measures or return on investment (ROI). That is, did the organization actually save money following the training program? For example, imagine that a bank trains its tellers to cross-sell Visa cards. The tellers rate the training session as being enjoyable, all of the employees pass a test on sales tecniques, and sales attempts increase by 30%. The bottom-line approach would then ask the question, If we spent $5,000 training the tellers how much more revenue was brought in as a result of the training? if the answer to the question is more than the amount spent on training, then the

End