Medium 9781576753484

Evaluating Training Programs

Views: 2319
Ratings: (0)
An updated edition of the bestselling classic
Donald Kirkpatrick is a true legend in the training field: he is a past president of ASTD, a member of Training magazine's "HRD Hall of Fame," and the recipient of the 2003 "Lifetime Achievement Award in Workplace Learning and Performance" from ASTD
In 1959 Donald Kirkpatrick developed a four-level model for evaluating training programs. Since then, the "Kirkpatrick Model" has become the most widely used approach to training evaluation in the corporate, government, and academic worlds.
Evaluating Training Programs provided the first comprehensive guide to Kirkpatrick's Four Level Model, along with detailed case studies of how the model is being used successfully in a wide range of programs and institutions. This new edition includes revisions and updates of the existing material plus new case studies that show the four-level model in action.
Going beyond just using simple reaction questionnaires to rate training programs, Kirkpatrick's model focuses on four areas for a more comprehensive approach to evaluation: Evaluating Reaction, Evaluating Learning, Evaluating Behavior, and Evaluating Results.
Evaluating Training Programs is a how-to book, designed for practitiners in the training field who plan, implement, and evaluate training programs. The author supplements principles and guidelines with numerous sample survey forms for each step of the process. For those who have planned and conducted many programs, as well as those who are new to the training and development field, this book is a handy reference guide that provides a practical and proven model for increasing training effectiveness through evaluation.
In the third edition of this classic bestseller, Kirkpatrick offers new forms and procedures for evaluating at all levels and several additional chapters about using balanced scorecards and "Managing Change Effectively." He also includes twelve new case studies from organizations that have been evaluated using one or more of the four levels--Caterpillar, Defense Acquisition University, Microsoft, IBM, Toyota, Nextel, The Regence Group, Denison University, and Pollack Learning Alliance.

List price: $47.95

Your Price: $35.96

You Save: 25%

Remix
Remove
 

28 Chapters

Format Buy Remix

Contents

PDF

 

1. Evaluating: Part of a Ten-Step Process

PDF

Chapter 1

Evaluating: Part of a

Ten-Step Process

he reason for evaluating is to determine the effectiveness of a training program.When the evaluation is done, we can hope that the results are positive and gratifying, both for those responsible for the program and for upper-level managers who will make decisions based on their evaluation of the program. Therefore, much thought and planning need to be given to the program itself to make sure that it is effective. Later chapters discuss the reasons for evaluating and supply descriptions, guidelines, and techniques for evaluating at the four levels.This chapter is devoted to suggestions for planning and implementing the program to ensure its effectiveness. More details can be found in my book Developing Managers and Team Leaders (Woburn,

MA: Butterworth Heinemann, 2001).

Each of the following factors should be carefully considered when planning and implementing an effective training program:

T

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

Determining needs

Setting objectives

 

2. Reasons for Evaluating

PDF

Chapter 2

Reasons for Evaluating

t a national conference of the National Society for Sales Training

Executives (NSSTE), J. P. Huller of Hobart Corporation presented a paper on “evaluation.” In the introduction, he says,“All managers, not just those of us in training, are concerned with their own and their department’s credibility. I want to be accepted by my company. I want to be trusted by my company. I want to be respected by my company. I want my company and my fellow managers to say,‘We need you.’”

“When you are accepted, trusted, respected, and needed, lots and lots of wonderful things happen:

A

Your budget requests are granted.

You keep your job. (You might even be promoted.)

Your staff keep their jobs.

The quality of your work improves.

Senior management listens to your advice.

You’re given more control.

“You sleep better, worry less, enjoy life more. . . . In short, it makes you happy.”

“Wonderful! But just how do we become accepted, trusted, respected, and needed? We do so by proving that we deserve to be accepted, trusted, respected, and needed. We do so by evaluating and reporting upon the worth of our training.”

 

3. The Four Levels: An Overview

PDF

Chapter 3

The Four Levels: An Overview

he four levels represent a sequence of ways to evaluate programs.

Each level is important and has an impact on the next level. As you move from one level to the next, the process becomes more difficult and time-consuming, but it also provides more valuable information. None of the levels should be bypassed simply to get to the level that the trainer considers the most important.These are the four levels:

T

Level 1—Reaction

Level 2—Learning

Level 3—Behavior

Level 4—Results

Reaction

As the word reaction implies, evaluation on this level measures how those who participate in the program react to it. I call it a measure of customer satisfaction. For many years, I conducted seminars, institutes, and conferences at the University of Wisconsin Management

Institute. Organizations paid a fee to send their people to these public programs. It is obvious that the reaction of participants was a measure of customer satisfaction. It is also obvious that reaction had to be favorable if we were to stay in business and attract new customers as well as get present customers to return to future programs.

 

4. Evaluating Reaction

PDF

Chapter 4

Evaluating Reaction

valuating reaction is the same thing as measuring customer satisfaction. If training is going to be effective, it is important that trainees react favorably to it. Otherwise, they will not be motivated to learn. Also, they will tell others of their reactions, and decisions to reduce or eliminate the program may be based on what they say.

Some trainers call the forms that are used for the evaluation of reaction happiness sheets. Although they say this in a critical or even cynical way, they are correct.These forms really are happiness sheets. But they are not worthless. They help us to determine how effective the program is and learn how it can be improved.

Measuring reaction is important for several reasons. First, it gives us valuable feedback that helps us to evaluate the program as well as comments and suggestions for improving future programs. Second, it tells trainees that the trainers are there to help them do their job better and that they need feedback to determine how effective they are.

 

5. Evaluating Learning

PDF

Chapter 5

Evaluating Learning

here are three things that instructors in a training program can teach: knowledge, skills, and attitudes. Measuring learning, therefore, means determining one or more of the following:

T

What knowledge was learned?

What skills were developed or improved?

What attitudes were changed?

It is important to measure learning because no change in behavior can be expected unless one or more of these learning objectives have been accomplished. Moreover, if we were to measure behavior change

(level 3) and not learning and if we found no change in behavior, the likely conclusion would be that no learning took place.This conclusion may be very erroneous. The reason no change in behavior was observed may be that the climate was preventing or discouraging, as described in Chapter 3. In these situations, learning may have taken place, and the learner may even have been anxious to change his or her behavior. But because his or her boss either prevented or discouraged the trainee from applying his or her learning on the job, no change in behavior took place.

 

6. Evaluating Behavior

PDF

Chapter 6

Evaluating Behavior

hat happens when trainees leave the classroom and return to their jobs? How much transfer of knowledge, skills, and attitudes occurs? That is what level 3 attempts to evaluate. In other words, what change in job behavior occurred because people attended a training program?

It is obvious that this question is more complicated and difficult to answer than evaluating at the first two levels. First, trainees cannot change their behavior until they have an opportunity to do so. For example, if you, the reader of this book, decide to use some of the principles and techniques that I have described, you must wait until you have a training program to evaluate. Likewise, if the training program is designed to teach a person how to conduct an effective performance appraisal interview, the trainee cannot apply the learning until an interview is held.

Second, it is impossible to predict when a change in behavior will occur. Even if a trainee has an opportunity to apply the learning, he or she may not do it immediately. In fact, change in behavior may occur at any time after the first opportunity, or it may never occur.

 

7. Evaluating Results

PDF

Chapter 7

Evaluating Results

ow comes the most important and perhaps the most difficult part of the process, you decide—determining what final results occurred because of attendance and participation in a training program.Trainers consider questions like these:

N

How much did quality improve because of the training program on total quality improvement that we have presented to all supervisors and managers? How much has it contributed to profits?

How much did productivity increase because we conducted a program on diversity in the workforce for all supervisors and managers?

What reduction did we get in turnover and scrap rate because we taught our foremen and supervisors to orient and train new employees?

How much has “management by walking around” improved the quality of work life?

What has been the result of all our programs on interpersonal communications and human relations?

How much has productivity increased and how much have costs been reduced because we have trained our employees to work in self-directed work teams?

 

8. Implementing the Four Levels

PDF

Chapter 8

Implementing the Four Levels

“E

verybody talks about it, but nobody does anything about it.”

When Mark Twain said this, he was talking about the weather. It also applies to evaluation—well, almost. My contacts with training professionals indicate that most use some form of reaction,“smile,” or

“happiness” sheets. Some of these sheets are, in my opinion, very good and provide helpful information that measures customer satisfaction.

Others do not meet the guidelines that I listed in Chapter 4. And many trainers ignore critical comments by saying, “Well, you can’t please everybody” or “I know who said that, and I am not surprised.”

Where do I start? What do I do first? These are typical questions from trainers who are convinced that evaluation is important but have done little about it.

My suggestion is to start at level 1 and proceed through the other levels as time and opportunity allow. Some trainers are anxious to get to level 3 or 4 right away because they think the first two aren’t as important. Don’t do it. Suppose, for example, that you evaluate at level

 

9. Managing Change

PDF

Chapter 9

Managing Change

here is one important ingredient that is basic to all evaluation approaches. There must be a realization that managing change is that ingredient. It starts with the determination of what changes are needed. We call it “determining needs.” We need to determine what knowledge, skills, and/or attitudes are needed to achieve the desired behavior and results. This means that training and development professionals must know the concepts, principles and techniques required for “managing” change. I have put “managing” in quotes because it has a twofold meaning. It not only means to decide on the changes to be made but also to get the acceptance of those involved in the change.

This chapter is written not only for training and development professionals but also for line managers. It is important to emphasize that the training and development professionals can control the determining of needs and the learning content. But it is also important to emphasize that changing behavior is under the control of the manager whose subordinates were trained.Therefore, these concepts, principles, and techniques are equally important to trainers and managers.

 

10. Using Balanced Scorecards to Transfer Learning to Behavior

PDF

Chapter 10

Using Balanced Scorecards to

Transfer Learning to Behavior

James D. Kirkpatrick, Ph.D.

I have asked my son, Jim, to write this chapter because of his knowledge and use of the Balanced Scorecard. In my description of moving from Learning to Behavior, I have concentrated on the motivation of the learners and the encouragement of their supervisors. I have described the various types of supervisors, including those who prevent or discourage the transfer. I have urged the trainers to work with the supervisors to help them become “encouraging” instead of “preventing” or “discouraging” bosses. I realize that this is not enough to be sure that the transfer takes place—hence this chapter on the Balanced Scorecard.

Don Kirkpatrick

believe that transferring learning to behavior is one of training’s biggest challenges. My father agrees—so much so that we recently wrote a book called Transferring Learning to Behavior: Using the Four

Levels to Improve Performance.The University of Toyota (UOT), under the leadership of Russ Mundi and Chuck O’Keefe, also believes it to be true. Transferring Learning to Behavior contains “ten best” practice case studies, one of which is from Toyota and in which Russ outlines a corporate challenge to improve a critical element of customer satisfaction. Based on customer feedback, the UOT designed a ten-step program to do just that.The program is designed to ensure that training participants actually apply (level 3) what they learned (level 2) during training.

 

11. So How Is E-Learning Different?

PDF

Chapter 11

So How Is E-Learning

Different? by William Horton

Many professional trainers are concerned with the evaluation of e-learning. No one is better able to provide answers than Bill Horton, who wrote the book Evaluating E-Learning, published by the American

Society For Training and Development (ASTD).While details for evaluating e-learning at all four levels are described in the book, this chapter sets forth the principles and approaches for doing so.

Don Kirkpatrick

Evaluating E-Learning Is the Same, But . . . ow well can an evaluation framework conceived in the 1950s apply to twenty-first century e-learning and its blended-, mobile-, and ubiquitous-learning variants? Back then computers weighed tons and the term “network” referred to television stations.

Yet, that four-level framework applies quite well.

Like all effective engineering models of evaluation it concerned itself solely with the results rather than the mechanisms used to accomplish those results.What we evaluate is not the artifacts or apparatus of learning but the outcome. The outcome of learning resides with the learners, not the pens, pencils, chalkboards, whiteboards, hardware, software, or other paraphernalia of learning.

 

12. Developing an Effective Level 1 Reaction Form: Duke Energy Corporation

PDF

Chapter 12

Developing an Effective Level 1

Reaction Form

Reaction forms come in all sizes and shapes.And the information generated may or may not be used to improve training programs.This case study describes a thorough process of developing a form to evaluate the significant aspects of the program. Emphasis is on items that relate directly to job performance and desired results.

Duke Energy Corporation

W. Derrick Allman,

Plan, Manage, and Procure Training Services,

Duke Energy Corporation, Charlotte, North Carolina

Duke Energy is a world leader in the development, collection, distribution, and production of energy-related services.The company conducts business in the global marketplace through national and international offices, having two primary corporate locations: Charlotte, North Carolina, and Houston, Texas. The company employs

23,000 individuals worldwide.

Evaluation processes at Duke Energy Corporation have taken many turns through the years. As we enter the era of aggressive competition in the energy services market, we are increasing our interest in determining the value that learning and development contribute to the business. An essential element in the valuation of learning and

 

13. Evaluating a Training Program for Nonexempt Employees: First Union National Bank

PDF

Evaluating a Training Program for Nonexempt Employees

125

solidations, employees have the pressures that all this change has brought to bear. CARE is a one-day program devoted to the bank’s largest population, the nonexempt employees who have shouldered major responsibilities throughout this growth cycle at First

Union.

CARE is an acronym for Communication, Awareness, Renewal, and Empowerment.The learning objectives are:

• Increase self-awareness by use of self-assessment tools and group feedback.

• Increase understanding of communication styles and develop

flexibility in one’s own communication style.

• Increase communication effectiveness by exposure to and practice in assertiveness concepts and skills.

• Understand and implement the steps of goal setting as a tool in career renewal.

Input from employee focus groups was instrumental in developing the course design.

The program is offered on an ongoing basis for new employees.

The majority of CARE I training occurred in 1991. More than

10,000 employees have attended CARE I.

 

14. Evaluating a Training Program on Developing Supervisory Skills: Management Institute, University of Wisconsin

PDF

Chapter 14

Evaluating a Training

Program on Developing

Supervisory Skills

This case study is based on a research project that was designed to measure changes in behavior and results.The program covered six topics and lasted for three days. Patterned interviews were conducted three months after the program with the participants and their immediate supervisors.

Management Institute,

University of Wisconsin

Donald L. Kirkpatrick,

Professor Emeritus

University of Wisconsin, Milwaukee

Developing Supervisory Skills, a three-day institute conducted by the

University of Wisconsin Management Institute, included six threehour sessions on the following topics: giving orders, training, appraising employee performance, preventing and handling grievances, making decisions, and initiating change. All the leaders were staff members of the University of Wisconsin Management Institute.

Teaching methods included lecture, guided discussion,“buzz” groups, role playing, case studies, supervisory inventories, and films and other visual aids.

 

15. Evaluating a Leadership Training Program: Gap Inc.

PDF

Chapter 15

Evaluating a Leadership

Training Program

This case illustrates an organized approach to evaluating a leadership training program at all four levels. Forms and procedures are included as well as the results of the evaluation.The approach can be adapted to any type of organization.

Gap Inc.

Don Kraft, Manager, Corporate Training

Gap Inc., San Bruno, California

Introduction: Why Leadership Training?

In 1994 the need for leadership training was identified for the storemanager level for the Gap, GapKids, Banana Republic, and International divisions of Gap Inc. The focus was on supervisory and leadership skills—how to influence and interact with store employees.

The program selected to meet this need was Leadership Training for

Supervisors (LTS). By providing store managers the opportunity to attend LTS, managers would not only improve their performance with supervisory and leadership skills, but job satisfaction would also increase.

As one manager shared after attending LTS, “This was the most rewarding experience I’ve had with the company in my four years as

 

16. Evaluating a Leadership Development Program: U.S. Geological Survey

PDF

170

Case Studies of Implementation

This case study provides an overview of the process and the findings of the first four years of investigation.

The USGS Leadership Program, directed by Nancy Driver as Program Manager, is a remarkable effort to enhance the organizational culture of the survey.A series of classroom experiences are offered for groups of about twenty-four participants at a time. The Leadership

101 course is a week-long, intensive workshop taught at the National

Conservation Training Center in Shepherdstown, WV, by a team of ten instructors in a series of modules designed specifically for the program.All but three of the instructors are USGS managers; the others are experts brought in to contribute particular segments. Participants are selected through an extensive nomination process, and there is a waiting list for admission to the course. Prior to the 101 class, all participants undergo a full 360-degree evaluation, completing a lengthy survey assessing their own behavior and requesting that a similar survey be completed by eight to ten of their co-workers. During the first days of the course, the relevant 360-degree feedback is compiled and distributed to each participant, and a buddy system is established to formulate action plans based on the comments provided from the surveys. Follow-up meetings by buddy pairs are required, and other modes of follow-up reinforce the classroom learning. The rest of the 101 week is devoted to a wide variety of leadership issues, including negotiation, supervision, team building, communication, and mentoring.

 

17. Evaluating a Leadership Development Program: Caterpillar, Inc.

PDF

Evaluating a Leadership Development Program

187

mize the near-term value of current assets but be prepared to make investments that take advantage of global opportunities.

This leadership challenge was not just to develop more leaders, it was to develop different leaders: leaders who epitomize collaboration, business acumen, and a global mind-set. Meeting this challenge to develop a new kind of leader also required new ways of thinking about leadership development.

Caterpillar has a rich history of growing its own leaders. In the

1970s and 1980s the annual management course at the Starved Rock

State Park in Illinois exposed leaders to the latest thinking about leading people and organizations.This course evolved into the Caterpillar

Advanced Management Program that prepared leaders to effectively expand Caterpillar’s business base.With the establishment of Caterpillar University and the College of Leadership in 2001, Caterpillar had an exciting new capability to develop leaders. Building a unified approach to leadership development across Caterpillar became the focus.

 

Load more


Details

Print Book
E-Books
Chapters

Format name
PDF
Encrypted
No
Sku
B000000023325
Isbn
9781576757963
File size
4.79 MB
Printing
Allowed
Copying
Allowed
Read aloud
Allowed
Format name
PDF
Encrypted
No
Printing
Allowed
Copying
Allowed
Read aloud
Allowed
Sku
In metadata
Isbn
In metadata
File size
In metadata