HOW MANAGERS CAN EVALUATE CLASSROOM TRAINING RESULTS

By:

Anthony C. Griffin and Adam K. Lang

1
 ABSTRACT

This article presents and discusses the Evaluation Meeting as a practical tool line managers can
use to assess the extent to which skills acquired through classroom training are being applied,
on-the-job. The Meeting, as introduced below, relies on a high degree of management
involvement and interest in training results. It is presented against the theoretical backdrop of
Kirkpatrick's four evaluation Levels, which are briefly examined by the authors. It is also a
format that permits managers and training professionals to gauge Level 2 and Level 3 impacts of
Kirkpatrick's model. The Evaluation Meeting showed results from training to be mixed, but
mostly positive. That is, the Meeting permitted company management to point to specific
improvements in shop floor communication, improved workflow, and improved delegation of
tasks and authority as outcomes of this training. Suggestions for follow-up and improvement of
the Evaluation Meeting are also discussed.

2
 HOW MANAGERS CAN EVALUATE CLASSROOM TRAINING RESULTS

How can managers know that classroom training has "paid off," that training objectives have
been achieved? Are managers satisfied with what training has accomplished?

The questions have dogged trainers, performance improvement professionals, and managers
since the earliest days of industrial adult education - that is to say, from about the turn of this
century. Kirkpatrick ( 1976, 1983), among others (Bramley, 1990; Brinkerhoff and Dressler,
1990; Clegg, 1987) have made Herculean efforts to provide tools to help answer these questions,
and the reader will certainly want to consult these evaluation gurus for detailed help on the
variety of means available to evaluate training effectiveness.

Among the most useful of these tools are the four evaluation levels proposed by Kirkpatrick (4,
5) who suggests that training can be evaluated at four different levels:

Level 4: Impacts of training on business performance.

Level 3: Application of classroom learning to on-the-job situations.
Level 2: Learning acquired by participants through training.
Level 1: Participant reactions to training, and the training environment.

In this model, each successive level provides more credible data that permit an objective
evaluation of training impacts. Also, as one climbs the evaluation "ladder," the evaluation effort
becomes more complex and more time-consuming. Levels 2 and 3 are the backdrops for the
following discussion. This article contends that the Evaluation Meeting can be an effective tool
for management to assess the extent to which new skills were acquired, and if these skills are
being used in work situations. A Level IV evaluation remains to be carried out.

This article discusses the “Evaluation Meeting” as a way that line managers can be directly
involved in evaluating training effectiveness and impacts. This is our focus because the author
had the opportunity of developing and presenting thirty-two hours of leadership skills training
for three key employees of a small manufacturing facility: a production manager, supervisor, and

3
lead person. The General Manager of this facility decided to follow-up on the classroom training
by conducting a series of these meetings, who thought this format to be an effective means to
determine if training was successful. The author embraced this idea because of its immediately
apparent advantages, namely:

(1) such a format clearly demonstrates top management interest in determining training success
and impacts, and did not have to be convinced of the need to support training efforts, as is
sometimes the case with line management;
(2) this manifestation of interest was clearly communicated to training participants, who were
plainly alert to management's interest in their performance during - and following -
classroom training.

The Evaluation Meeting is fairly straightforward in both design and execution. Its purpose is to
help management – in this case, the General Manager – understand what and how much was
learned by participants. Thus, the Meeting is most clearly seen as a means to gather data that
assess learning acquired through training: that is, Level II evaluation data.

However, this Meeting is also an opportunity to evaluate impacts at Level III – that is, on-the-job
application. This will be seen from the discussion below.

Background to the Project

Ceram-Tek manufactures ceramic parts and accessories, and is located in Corona, California. A
subsidiary of Homexx International, Inc., it is the major supplier to Homexx of such ceramic
products as bath and kitchen fixtures. Homexx, in turn, sells to its customers who are principally
in the kitchen and bath industry.

Ceram-Tek numbers about twenty full- and part-time employees, working in two shifts, and is
led by the General Manager. Reporting to him is a Production Manager, Supervisor and Lead
person. None of the three trainees had participated in any prior leadership skills training.
Training was conducted in Spanish because it was the first language of all three participants.

4
Training materials were also in Spanish. The average educational level of the participants was
estimated to be < 6th Grade, in Mexico, which means that students are functionally literate and
are able to read and write, but somewhat below their grade level. With the exception of one
participant, none had participated in any formal skills training.

The Training Design: Classroom plus Coaching

The goals of this training were to equip participants to identify and resolve interpersonal and
technical problems, manage conflicts and disagreements more effectively, and to communicate
more clearly with subordinates, with each other, and with management. Of the thirty-two hours
of this program, twenty of these were classroom training. This was complemented with twelve
hours of one-on-one coaching of trainees. The purpose of this coaching feature was to facilitate
transition of skills acquired in the classroom, to shop floor application. Coaching was structured
so that each trainee met with the instructor for one hour, once per week, for a total of twelve
coaching hours. Coaching sessions were structured around the training objectives for that
particular week, e.g., how to apply the four basic leadership styles that were being taught, how to
use active listening skills, etc. and personalized to the particular learning needs of each
participant. The author also submitted recommendations on how to reinforce training with
participants, in their daily work.

Gauging Training Results: The Evaluation Meeting

Towards the end of training, the General Manager told the participants that he wanted to meet
with them after training to determine how much they had learned as a result of training. He
asked the three trainees to review their training and to be prepared to present individual thirty-
minute summaries of key ideas or skills acquired during training, in a meeting to be held two
weeks hence. He asked participants to use that time to discuss and decide among themselves
which topic and skills areas each person would present, and offered to provide them the means to
prepare for their presentations, e.g., help in preparing an outline for their presentation, providing
them with appropriate supplies and administrative support.

5
About a week later, in informal discussions between the General Manager and training
participants, the General Manager concluded that the three employees had not agreed about how
to make their presentations. To provide direction and structure, he asked each person to refer to
their training manuals, and divide them among themselves by subject matter so that each
participant presented approximately one-third of the training content. Participants agreed to do
this. In the interim, the General Manager continued to make himself available to the three
employees, in the event they needed help in preparing their presentations. The Meeting was held
two weeks following training, and lasted about ninety minutes.

There were three presentations during this Evaluation Meeting. The first trainee presentation
covered Modules 1-4 of an eight-module training program, namely: "Interpersonal
Communication," "Leadership," and "Conflict-Resolution," and "Delegation." The second
presentation accounted for Modules 5-8 of training: "Motivating and Disciplining," "Problem-
Solving," Train-the-Trainer Skills," and "Legal Issues." The third and last presentation dealt
exclusively with the how to use the Ishikawa Diagram to solve work problems.

During and following each presentation, and to test learning, the General Manager asked
questions; e.g., " What was the most important skill you learned?" "What are the most common
conflict-resolution styles that you see us using here at Ceram-Tek?" "How do you use the four
basic communication styles (taught in training) with the people you supervise?"

Overall results were mixed, but mostly positive. According to the General Manager, the
employees reported that the most useful skills acquired were interpersonal ones. For instance,
training reportedly enhanced participant's ability to communicate and listen. One Level 3
evidence of improved communication at work was less time spent by the Manager and the three
employees in locating materials, messages, and equipment. Another evidence is that some
workflow improvements have resulted from improved communication in message flow and
reduced lost time. For instance, subsequent to training, the General Manager instituted thirty-
minute overlap meetings between Day and Night shift Supervisors. Previously, shift overlap
lasted no more than five minutes, which resulted in unclear, confusing, or missed
communication opportunities. With increased overlap, Day and Night shift problems related to

6
scheduling, problems with machine operation, and personnel issues were minimized, and
resulted in shortened lines of communication and a smoother "handover" of work shifts.

A third evidence to support the conclusion that some classroom learning was being applied to
workplace situations was the institution of Monday morning meetings between the General
Manager, the Production Manager, the Supervisor and the Lead, to review production and other
issues for the upcoming week, and to anticipate possible problems. Prior to training, these
meetings were not held, or -when held - did not always include all four key personnel. Meeting
results were recorded and used to improve productivity, workflow, communication, etc.?

Concurrent with classroom training and coaching, was training the Supervisor and Lead on how
to use a new e-mail system. All three personnel have adopted use of this system, which -
according to the General Manager - has further contributed to clearer and more opportune
communication among them. Clearer communication, in this case, was measured by a reduction
in the number of machine, personnel, and labor scheduling problems that was the norm before
training and which resulted from the sporadic and informal communication between the
Supervisor and Leads. Improved communication between these employees and the General
Manager has resulted in a "just-in-time" process for scheduling of temporary labor, and more
timely scheduling for machine maintenance and tooling.

The Evaluation Meeting demonstrated that the trainees had gained some measure of skill in
using the Ishikawa Diagram to solve problems. The Meeting satisfactorily demonstrated that all
three had acquired the basic tools associated with such other training Modules as "Delegation,"
"Conflict-Resolution," "Motivating and Disciplining," and "Legal Issues."

The principal author submitted a report at the conclusion of training, with recommendations for
management action. Two of recommendations were: (1) collaborative development of job
descriptions that define main duties, tasks and responsibilities of the Production Manager, the
Supervisor and Lead; (2) developing performance and career development goals for the three
trainees. Both of these recommendations have been implemented: job descriptions have been
developed and, while the General Manager has initiated career-planning discussions with his
three key personnel, more work remains to be done in this area. Additional recommendations to:

7
(3) formally train these personnel in basic Quality tools, (4) systematically plan toward
development of work teams, and to (5) formally train temporary employees and other shop floor
personnel, are either under consideration by management, are in the process of being
implemented, or as yet have not generated any Level 2 or Level 3 evidences.

How the Process Can be Improved

Both authors agree that the Evaluation Meeting was a useful and reliable way to gauge and
assess the overall "value" of training to the Ceram-Tek organization; that is, the Meeting enabled
management to reasonably conclude that training was worth the investment. At the same time,
both agree that any future Evaluation Meetings should reflect the following improvements:
Participants should have the option of making at least part of their presentations in
1.
Spanish, provided they have adequate time to prepare and that they can choose which
portions to present, and in which language.
Participants need to be more formally prepared to make their presentations. This
2.
preparation should include coaching on how to prepare a presentation, selecting
appropriate presentation media, e.g., overhead? flipchart? etc., how to prepare a
written presentation outline, and a presentation rehearsal. This preparation requires an
investment of time if the Meeting is to yield desired results.
There was little prior discussion among all parties about what each expected from the
3.
Meeting, and no written agenda. Trainee participants and management should discuss
and agree on what each expects from such a Meeting. A Meeting agenda should
reflect these expectations, and be shared beforehand among all attendees.
Document results and conclusions from the Meeting. This documentation can then be
4.
used to plan further Meetings, and as templates for discussion between management
and key personnel about career development needs, including possible future skills
training for trainees.

8
 One-on-one coaching between the Instructor and individual participants was a feature
5.
of this training intervention. However, the contribution of coaching to the overall
effectiveness of this training effort is unclear. This is partly because the Evaluation
Meeting did not directly incorporate learnings from the coaching sessions;
consequently, there was no format to evaluate the extent to which coaching did or did
not supplement the classroom activities. Future coaching efforts could be improved
by more formally and explicitly linking coaching to classroom work, and by using the
Evaluation Meeting to assess coaching's relationship to classroom accomplishments.

Conclusions

Management concluded that the Evaluation Meeting is a cost-effective way to assess training
impacts at three of Kirkpatrick's evaluation levels. It permits management to draw conclusions
about the value of training to the organization, and is also a clear demonstration of management
interest in training. It requires a formal agenda and investment of organization time in preparing
trainees to present key learnings. Its potential pitfalls, as discussed above, can be anticipated and
avoided, with proper use of meeting planning skills.

The Meeting clearly communicates management interest and involvement to training

participants. In the above case, participants plainly saw the need to apply classroom skills
training to work on the shop floor. This was largely driven by management requiring them to
review training material, and make formal presentations to management, at the conclusion of
training. The Meeting, therefore, is a means to draw some conclusions about (1) the skills
participants acquired, through training, and; (2) the degree to which participants are using these
acquired skills in their work. Management reached these conclusions by relying on "evidences;"
for example, improvements in workflow, shortened communication lines, and improved
delegation practices.

9
Author Biographies

Anthony Griffin has delivered human resource development and performance improvement
services through his company, Teamworks, since 1994. He specializes in training and consulting
interventions for employers with large numbers of Latino employees. He can be reached at:
training@teamworks1.com. Phone: (909) 784-9330. Fax: (909) 784-5003. His website is:
http://www.teamworks1.com.

Adam Lang has served as General Manager of Ceram-Tek since 1995. Prior to coming to
Ceram-Tek, he worked as a ceramic engineer, and in marketing, for Sylvania, He has a B.S. in
Ceramic Engineering from Alfred University, and an M.B.A. from Pepperdine University. He
serves on the Executive Committee of the Association of American Ceramic Component
Manufacturers, and is a member of the American Ceramic Society. He can be reached via e-mail
at: adam.lang@ceramtek.com or at www.ceramtek.com.

References

(1) Bramley, Peter (1990). Evaluating Training Effectiveness. Berkshire: McGraw-Hill.

(2) Brinkerhoff, R.O. and Dressler, D.E. (1990). Productivity Measurement: A Guide for
Managers and Evaluators. Thousand Oaks: Sage.

(3) Clegg, William (1987, February). "Management Training Evaluation - an Update."

Training and Development Journal, pp. 65-71.

(4) Kirkpatrick, Donald (1983, November) "Four Steps to Measuring Training

Effectiveness." Personnel Administrator, pp. 19-25.

10
(5) Kirkpatrick, Donald L. (1976) "Evaluation in Training." In Robert L. Craig (Ed.),
Training and Development Handbook, 2nd ed. (Chap. 18), New York, NY, McGraw-Hill.

11