Understanding the Buzz Around E-Learning:

“Searching for Faster/Better/Cheaper Learning – Effectiveness of E-Learning Techniques”

Roderick A. Munro, Ph.D.
Business Improvement Coach
RAM Q Universe, Inc
1135 Terminal Way, Suite 209
Reno, NV 89502
rodmunro@starband.net
www.ramquniverse.com

SUMMARY

The mantra “fast-better-cheaper” is being applied to learning interventions and one method of learning that is gaining
popularity is called E-Learning. This presentation takes a basic look at E-Learning and gives the participant a fundamental
understanding of what it is and how to evaluate the potential effectiveness of it.
Some of the questions reviewed in this paper and included in the presentation:
• What is E-Learning?
• What is all the hype that is being bandied about around this learning technique?
• What do you need to know about setting up or buying E-Learning?
• How do you judge the different learning styles in E-Learning?
• How might you evaluate the effectiveness of E-Learning?

INTRODUCTION

Have you noticed the number of college diplomas that are now being offered through the Internet? One college in
the Detroit, MI area is now advertising on the radio to try to get students to come back for “real” training in a traditional
college with classrooms, live sporting events, live faculty members, a real library and even to have a real mascot.

Many companies are also cutting down on the number of live on-site classes that are being held (except maybe for
Six Sigma training) in favor of employees taking classes on either the intranet or internet. What has been happening to the
way people learn over the last 50 plus years is an interesting study of our society. After World War II, the German High
Command comment that the only reason that they lost the war was the they had underestimated the ability of the United
States of America in its ability to train vast numbers of people to do what was needed (from solders to factory worker) to
produce the armaments need to wage war (see figure 1).

From those very successful days of training during WWII up through today, the learning process as progressed from
an instructor lead activity to a computer lead activity. Tie this with the intranet or internet and you have the basics of e-
Learning. Some of the many electronic offerings you can find on the web today include:
• Macromedia Breeze
 E-learning: Accelerating Web communication, collaboration and training is a Breeze....
www.macromedia.com
• E-Learning
Work towards a Bachelor of Science or master's degree online at University of...
degrees.uofphx.info
• University Alliance - E-Learning Degrees
U.S. residents, earn your MBA, BA (business, CIS, MIS, accounting, CJ, RN-to-BSN) or...
www.universityalliance.com
• E-Learning
Hosted elearning solution to teach and train. Request a free demo.
www.cyber-grad.com
• Online PC, Unix Labs for E-learning
Add live hands-on labs to your online technical training classes. Hatsize TrueLab...
www.hatsize.com
• Online Degree Programs
Business, IT, Education, Criminal Justice and Healthcare programs from accredited...
www.classesusa.com
• Online Degree Programs
Kaplan University Official Site offers online programs in criminal justice, business,...
kaplan.edu
• eLearning Courses
Corporate and government solutions that include a Personal Learning Service for...
corporate.mindleaders.com

WHAT IS ALL THE BUZZ ABOUT?

Research that has been done by the National Training Laboratories (NTL) Institute for Applied Behavioral Sciences
in Alexandria, VA suggests that during a learning intervention we: retain five (5) percent of what we hear, 10 percent of what
we read, 20 percent of what we see and hear (as in a lecture), 75 percent of what we practice and 80 percent if we teach other
and immediately use ourselves (Boehle 2005). Our own experiences as well as many formal studies indicate that we will
learn something much quicker if we are interested in and are able to apply what we are learning in real time, real life
situations.
During the 1980’s and 1990’s a series of articles called the Clark-Kozma Debates were published that examined, is
it the method or is it the technology that really matters in a person’s ability to learn! The jury may still be out.
Article 1: Reconsidering Research on Learning from Media: Richard E. Clark, 1983
Clark’s initial article on media covers the research that has been done on whether or not a given medium has had influence on
learning outcomes. The article states:
⇒ Studies have shown that media does not influence learning under any conditions. Studies that did indicate a positive
result were because the studies themselves were flawed.
⇒ His most famous statement: “The best current evidence is that media are mere vehicles that deliver instruction, but do
not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition.
Basically, the choice of vehicle might influence the cost or extent of distributing instruction, but only the content of the
vehicle can influence achievement.” (P.445)
⇒ Clark states that studies were not about media, they were research with media. Media were not the focus of studies,
merely the conveyances for information.
⇒ He cites studies by Glaser and Cooley that recommend using any acceptable medium rather than expecting the medium
to make a difference in learning.
⇒ The rest of the article focuses on the method used to convey learning as being significant, not the medium. He states that
the method may be associated with a medium, but the medium is not to be the focus.
Clark states that it doesn’t make sense to do any more media studies or comparison research. He does support his own
research that covers attributes of the media as having effect on learning. He says that this will have impact on instructional
design, but not on instructional theory. The attributes can facilitate learning, but there are instances when students didn’t like
a certain medium, but scored better with it than with other media.
He summarizes: “ The point is made, therefore, that all current reviews of media comparison studies suggest that we will
not find learning differences that can be unambiguously attributed to any medium of instruction.

Kozma Returns Fire in Learning with Media, 1991

 One may wonder why it took Kozma until 1991 to rebut Clark, but he does it. Here are his arguments.
⇒ His definitive statement: “The research review in this article suggests that capabilities of a particular medium, in
conjunctions with methods that take advantage of these capabilities, interact with and influence the way learners
represent and process information and may result in more of different learning when one medium is compared to
another... .
⇒ He differentiates between internal and external cognitive environments (you still have time to sign up for truck driving
school and skip this stuff)
One of the confusions today may be in the definitions that we use for learning and training. Rosenberg (2003) defines
learning as “a process by which we take in information and translate it into knowledge or skills.” He defines training as “an
external experience we go through.” These definition tend to support Clark debate that the nutrition of the groceries that are
delivered (learning) is more important than the vehicle (training) that does the actual delivery.

WHAT IS E-LEARNING?

E-Learning means different things to different people. Many call this web-based, computer, internet, intranet, network,
etc. training (see table 1). Most people do agree on that fact that E-Learning has some component of electronic device to
assist in the learning process. For the purposes of this paper, we will include all forms of electronic computer devices that
allow the learner to proceed in a self-paced format.
Using this definition, we can include anything from E-Books to full functional multimedia presentations that can be used
as an on-demand or group format. With new technologies being developed almost daily, the future use of technology may
very well be different for many types of applications.

Types Of Media And Their Equipment: (from simple to progressively complex)

• Print
- Manuals (E.G. Instructor, Participant, Procedural Etc.) - Workbooks
- Job Aids (E.G. Checklists, Flowcharts, Diagrams) - Wall Charts, Posters, Flipcharts
• Electronic Print – simple page turning
• Hypertext Documents - PC
• Traditional Classroom (Platform)
- Overheads and slides - Instructor - Video and Audiotapes - Training Facility
• Hypermedia Documents
• Satellite Distance Learning (SDL)
• Pic-Tel Teleconference
• Interactive Keypads And Whiteboards
• Desktop Videoconference
• Kiosk
• Computer Based Training (CBT)
• Interactive Multimedia (CD-Rom)
- PC with Audio Board, Speakers, Headset And CD-Rom Drive
• CD-Internet Hybrid Program - Multimedia PC with Internet
• Virtual Reality
• Intranet or Internet
• Electronic Performance Support System (EPSS)
• User Groups/Bulletin Boards – PC with Internet
• Expert System
• Teleconference
• Webinar
• Simulations (Boehle 2005)
Table 1: Web Definitions of E-Learning (Google Search)

Tip: Try Google Answers for help from expert researchers

Definitions of e-learning on the Web:
• Education via the Internet, network, or standalone computer. Network-enabled transfer of skills and knowledge.
e-learning refers to using electronic applications and processes to learn. e-learning applications and processes
include Web-based learning, computer-based learning, virtual classrooms, and digital collaboration. Content is
delivered via the Internet, intranet/extranet, audio or video tape, satellite TV, and CD-ROM.
www.learnframe.com/aboutelearning/glossary.asp
• Education offered using electronic delivery methods such as CD-ROMs, video conferencing, websites and e-
mail. Often used in distance learning programmes.
www.mba.hobsons.com/sections/study_guides/GlobalGlossary/glossary.htm
• The delivery of content, via all electronic media, including the internet, intranets, extranets, satellite, broadcast,
video, interactive TV and CD Rom. E-learning encompasses all learning undertaken, whether formal or
informal, through electronic delivery. www.ltsnhsap.kcl.ac.uk/site/resources/glossarykeywords.htm
• The component of distributed learning that includes digital content, is experienced through a technology
interface, and is Internet-enabled. Collaboration is a desirable feature but not a requirement. e-marketplace
(electronic marketplace) A Web site that enables buyers to select from many suppliers. E-marketplaces - which
focus on putting the buyer in control - are buying environments that aggregate supplier content and provide
decision support tools that enable a buyer to make the most informed decision. www.i-
stt.com/resources/glossary/E.html
• Covers a wide set of applications and processes such as web-based learning, computer-based learning, virtual
classrooms, and digital collaboration. It includes the delivery of content via Internet, intranet/extranet, audio and
videotape, satellite, and CD-ROM. However, many organizations only consider it as a network-enabled transfer
of skills and knowledge.
www.neiu.edu/~dbehrlic/hrd408/glossary.htm
• Any technologically mediated learning using computers whether from a distance or in face to face classroom
setting (computer assisted learning).
www.usd.edu/library/instruction/glossary.shtml
• While the number of possible definitions for e-learning is considerable, in this study e-learning includes
learning that has an electronic component in its delivery. For instance online learning or distributed learning
where email or videoconferencing or digital formats are used. While it is considered by some to mean
specifically learning across electronic networks, this study has taken the more general and widely used
application of the term for ease of understanding. www.bbk.ac.uk/ccs/elearn/glossary.htm
• Broad definition of the field of using technology to deliver learning and training programs. Typically used to
describe media such as CD-ROM, Internet, Intranet, wireless and mobile learning. Some include Knowledge
Management as a form of e-learning. Took awhile for the right term to come about, circa 1995 it was all called
"Internet based Training", then "Web-based Training" (to clarify that delivery could be on the Inter- or Intra-
net), then "Online Learning" and finally e-learning, adopting the in vogue use of "e-" during the dot com boom.
The "e-" breakthrough enabled the industry to raise hundreds of millions from venture capitalists who would
invest in any industry that started with this magic letter. e-learningguru.com/gloss.htm
• Covers a wide set of applications and processes, such as Web-based learning, computer-based learning, virtual
classrooms, and digital collaboration. It includes the delivery of content via Internet, intranet/extranet
(LAN/WAN), audio- and videotape, satellite broadcast, interactive TV, and CD-ROM.
www3.sympatico.ca/krewski/mde615groupc/glossary.html
• Electronic learning or learning via Information and Communication technologies (ICTs) on the analogy with e-
mail. But this interpretation could be contested (see Woodall, 2001) www.windeatt.f2s.com/ijet/glossary.htm
• –refers to any type of learning that uses electronic media. It means many different things to people but is a
popular media buzz word and should be avoided.
www.plymouth.edu/psc/infotech/tlc/itcc/Glossary.htm
• Network-enabled transfer of skills and knowledge. www.oslerbooks.com/telecom/cisco.eglossary.html
• On-line educational opportunity; i.e., distance learning
www.irvingisd.net/~mvillarreal/M.%20Villarreal's%20Glossary.htm
• The learning byproducts from the marriage of the Internet and education. The Internet has transformed the way
education occurs and creates new ways of learning.
www.optasia.com/opguide/elearngloss.htm
Benefits

Some of the benefits of e-Learning have been described at http://www.internettime.com/itimegroup/more.htm

"eLearning" is a vision of what corporate training can become. Its hallmarks are:
• learning on Internet-age steroids: often real-time, 24/7, anywhere, anytime
• learner-centered, personalized to the individual & customized to the organization
• network-assisted, often assembling learning experiences on the fly
• a blend of learning methods -- virtual classroom, simulation, collaboration, community, even classroom...
• the whole learning enchilada, from assessment through testing and sometimes certification
• online administration -- handling registration, payment and charge-backs, and monitoring learner progress

Detractors

On the other hand, there are many people who still do not like to use electronic means of learning. Some of the
things they raise include:
• Costs to much
• Takes too much time
• No-one here know how to make this stuff
• Has not worked before
• Does not work on our computers

WHAT IS ALL THE HYPE THAT IS BEING BANDIED ABOUT AROUND THIS LEARNING TECHNIQUE?

A mantra of sorts has developed in industry around staying in business today – that an organization has to be
faster/better/cheaper than their competition. The same thing can be said about training and the learning process. There are
still appropriate times when stand up trainers/consultants are the best way for people to learn, however, with every new
development in electronic technologies, there are now many ways people can learn.

Faster

Is the learning available when it is actually needed? The old traditional stand-up training was usually conducted
only when there were enough participants to make the delivery worthwhile from a cost standpoint. This meant that many
prospective learners had to wait longer periods of time, in some cases, to attend the training if at all. Using electronic devices
can sometimes allow for on demand training/learning interventions, which becomes the ideal situation. Many people today,
can simply surf the web for ideas or learning to meet the needs of the task/skill that they are trying to acquire.

Better

Defining what better means can be a tricky topic (see section on Training Evaluation). Here the reader will need to
think about what was needed to start with and how well did the learning intervention actually deliver the skills, knowledge,
information, etc that the participants need on the job. Better in some cases might mean that the participant learned more that
using other techniques for learning. In other cases, the participant may have to demonstrate a skill that can be accomplished
with a certain level of proficiency. Being clear up front in what is meant here will assist the purchaser or manager of e-
Learning in really getting what is really needed for the money that is being spent.

Cheaper

The organization’s management will need to look at both the up front costs of the learning intervention and the costs
of ensuring that learning was effective. This has been a problem in the past when training was conducted, but either the
effectiveness was not evaluated or the organizations culture was not changed to match up with the new skills that were
expected to be used by the people. Thus the up front costs may not be the cheapest in the long run.

By putting these three together, the e-Learning needs to be flexible and ready to meet the needs of the perspective
learner. One study is Ten Ways Online Education Matches, or Surpasses, Face-to-Face Learning
(http://ts.mivu.org/default.asp?show=article&id=1059) by Mark Kassop. In his own words, M. Kassop layouts out his basic
ideas of where he thinks things are headed:

“Can online courses match traditional face-to-face (F2F) courses in academic quality and rigor? Can online courses
achieve the same learning objectives as F2F courses? Can students learn as much and as well online as they do in F2F
courses? Not only is the answer to these questions a resounding "yes," but there are many ways that online courses may
actually surpass traditional F2F classes in quality and rigor. For the record, I am writing from the perspective of a long-time
(30+ years) classroom instructor, a sociology professor, a day-to-day user of the consortium's WebCT course management
system, and a person who truly thrives on interaction with students. I still thoroughly enjoy the classroom environment, and I
believe that it is a sound arena for teaching and learning; however, as this article will suggest, I am distinctly impressed with
the early successes and potential of the online environment as a teaching/learning medium.

1. Student-centered learning:
2. Writing intensity:
3. Highly interactive discussions:
4. Geared to lifelong learning:
5. Enriched course materials:
6. On-demand interaction and support services:
7. Immediate feedback:
8. Flexibility:
9. An intimate community of learners:
10. Faculty development and rejuvenation:”

WHAT DO YOU NEED TO KNOW ABOUT SETTING UP OR BUYING E-LEARNING?

Instructional Theory – Profound Knowledge

Dr. Deming used the term Profound Knowledge to describe what an organization or management team needed to
ensure that they used the skills of the organization in the most optimal manner. From a training or e-Learning perspective,
this is called Instructional Theory. The developers of the formal e-Learning intervention should be aware of what the
learners needs are in order to create the desired outcomes of the organization. E. Rice in the Instructional Technology or
(Educational Technology) Masters and Doctoral Study and Reference Deck developed the following table:

Learner Characteristics Design Procedure for New Learning

Intellectual skills Stimulate retrieval of: (1) prerequisite skills as components of a new skill; (2)
subordinate skills essential to cognitive strategies; and (3) basic skills involved
in verbal information learning, attitude learning, and motor skill learning.
Cognitive strategies Provide for retrieval when available.
Verbal information Stimulate recall of propositions that may cure retrieval of newly learned
intellectual skills. Provide retrieval of a meaningful context (schema) for new
learning of verbal information. Stimulate retrieval of situational content for
attitude learning.
Attitudes Activate for learning motivation.
Motor skills Recall essential part skills
Schemas Activate retrieval of schemas consisting of complex networks of propositions
to aid new learning of intellectual skills, cognitive strategies, verbal
information, attitudes, motor skills.
Abilities Adapt instruction to differences in ability whenever possible.
Traits Adapt instruction to learner differences in traits when possible.

Human Performance Technology

According to the Handbook of Human Performance Technology, Human Performance Technology (HPT) “is a field
of endeavor that seeks to bring about changes to a system, in such a way that the system is improved in terms of the
achievements it values.” HPT is a field of study that the designers of formal e-Learning should be aware of and use in the
creation/development of the learning intervention. Otherwise, how will the purchaser of the programs have any assurance
that the needs of your organization will in fact be met by the proposed learning intervention?

Instructional System Design - Advanced Quality Planning

Instructional System Design (ISD) is the professional training developers versions of what quality professionals call
Advanced Quality Planning (AQP). There are several models of ISD that have been developed over the years, but the all use
a systematic way of identifying what the learning audience needs, develops the learning intervention, delivers the learning
process and usually has a method of evaluating the results.
As found in the Instructional Technology or (Educational Technology) Masters and Doctoral Study and Reference
Deck, E. Rice states, “The ISD Model is a systems approach for the orderly and comprehensive design, development, and
management of both instructional materials and instructional systems. It can be traced back to the fields of psychology and
communications, with foundations in learning theory, systems engineering, instructional technology and empirical research.
McCombs, 1986. ISD is directly traceable to general systems theory, which is its foundation.

Dick (1981) states that it was from consensus of Andrews and Goodson that a generic model emerged which included
descriptive and prescriptive sequences of steps whose execution resulted in predictable learning outcomes. The model from
Dick and Goodson has 10 generic steps.
1. Needs assessment
2. Specification of broad goals and detailed objectives of learning outcomes
3. Development of criterion-referenced tests for assessing goals and objectives
4. Analysis of goals and objectives to determine types and sequencing of skills
5. Analysis of learner characteristics
6. Specification of instructional strategies based on task and learner analysis
7. Selection of media to implement strategies
8. Development of courseware
9. Formative evaluation and revision of course materials
10. Installation and maintenance of programs.”

HOW DO YOU JUDGE THE DIFFERENT LEARNING STYLES IN E-LEARNING?

There are numerous learning styles that can be considered when designing/developing e-Learning interventions.
The classic foundations of learning theories are summarized here from E. Rice in the Instructional Technology or
(Educational Technology) Masters and Doctoral Study and Reference Deck:

“Ivan P. Pavlov (1849-1936) made dogs salivate. His studies included causing a given response by an unrelated stimulus
(also called unconditioned stimulus). His theory is called Classical Conditioning. There are four facets to classical
conditioning.
1. Reinforcement: It is a situation in which a conditioned stimulus is repeatedly followed by an unconditioned stimulus
and its natural response. Note that it is also possible to obtain different responses to two stimuli by using reinforcement
techniques. This is differentiation.
2. Extinction: is achieved by dropping the unconditioned stimulus out of the experiment until the conditioned response is
no longer made. The conditioned response, however, can suddenly reappear and this is called spontaneous recovery.
3. Inhibition: Inhibition can occur in the form of experimental extinction, or because of confusing stimulus. It can occur
as a result of differentiation, a situation in which the subject distinguishes between two stimuli, which previously
generated the same response.
Generalization: It occurs when the eliciting properties of one stimulus are taken on by another stimulus with which it is
paired. If both stimuli are reinforced, then generalization can occurs. If they are not both reinforced, differentiation

B.F. Skinner’s theory is called Operant Conditioning. The theory concerns operants, which are behaviors that
occur without any apparent stimulus. By reinforcing these behaviors, he was able to condition behavior into happening. He
waits for a chance behavior to occur, then reinforces it. The strength of the operant is determined by how high the probability
is that the behavior will be repeated.

Skinner differentiates between positive and negative reinforcers. Positive reinforcement is a stimulus, which will
increase the operant strength when it is added to a situation. Negative reinforcement makes the desired behavior more likely
to occur because it is removed from the situation. Punishment is not a negative reinforcer because it is added to the situation.
Punishment should be used intermittently, if at all.

Schedules for reinforcement have a great effect on the conditioning, strength of responses, and extinction. Shaping
refers to changing behavior gradually by reinforcing the approximate behavior until it is shaped into the actual, desired
behavior.

E. L. Thorndike (1874-1949). His theory was called Connectionism. He saw learning as a trial and error process. You
learn by selecting a response to a stimulus, thus making a “connection”. There are three major “laws” to the theory.
1. The Law of Effect states that once a connection is made, the strength of that connection is dependent on what follows. A
reward will strengthen the behavior, making it like a habit, and punishment will weaken the behavior. Thorndike stated
that rewards were much more important than punishments.
2. The Law of Readiness indicated that if an organism has a state of readiness, making a connection will be satisfying, and
the animal will do things to maintain that connection. If the organism is not ready, the connection will become
annoying, and the animal will do things to eliminate it. It is a preparation for action, and has nothing to do with
prerequisite skills.
3. The Law of Exercise states that connections are strengthened through practice and weaken through disuse. He
emphasized the importance of not just doing simple practice, but practice that is followed by rewards.

The principle concerns of cognitive learning theory are how people remember and retrieve information from memory.
Unlike behaviorists, cognitivists have studied humans to derive their theory. Gagné and White as quoted by Richey state that
cognitive theories support the basic notion that the effects of instruction may best be understood by exploring the three-term
relation Instruction-Memory Structure-Learning Outcome.
• Programming of a new rule for information processing
• Types: short term memory, long term memory, short term sensory storage
• Processes are: thinking, problem solving, language, concept formation, information processing
• Concerned not so much with what learners DO - but WHAT they KNOW and HOW they acquire it
• ID implications are: Active involvement with the learner, learning hierarchy, structuring of information, creation of
learning environment
• Make knowledge meaningful
• Instruction must be based on existing schema of student
• Some cognitive strategies supported by dialogue between teacher and students as they attempt to gain meaning:
− Summarization − prediction − sequencing
− question generation − chunking − advanced organizers
− clarification − loci − flowchart for schema”

Most people learn best using a combination of the following styles: Auditory, Reading, Visual or Kinesthetic. A
well-designed e-Learning initiative will use all four styles and consider the items discussed in this section to help ensure that
participants are engaged and will enjoy the learning experience.

HOW MIGHT YOU EVALUATE THE EFFECTIVENESS OF E-LEARNING?”

Many people are aware of using a survey at the end of learning event. This is what is called the Level 1 evaluation
and is part of the Kirkpatrick Evaluation Model (see further down). However, many evaluation processes are very weak and
only see to get a quick snapshot of what a participant thinks about what they just experienced without looking at the long-
term use of the new skills that were to be learned back on the job. In thinking about evaluation of e-Learning, we might first
look at what measures might be reviewed. The eLearning Guide
(http://www.elearningguild.com/pdf/1/values_survey_results_-_final.pdf) – survey series December 2003: shows the value of
e-Learning (NOTE: only 48 responses to the survey taken Oct-Dec 2003):

3. For the measures that are most critical to your organization, what data source is used for this information? (Provide data
sources for the 3 most critical measures.) The following data sources were listed by respondents for each of the measures
listed:
Measures Data sources listed by respondents

Learner measures
Learning gains Tests
Learner performance measures Quizzes and tests, surveys, performance goals

Course measures
Course enrollments
Course completion rates
Course revenue or profits Course revenue or profits
Customer satisfaction with specific course
Customer satisfaction with curriculum offerings
Increased customer demand for e-Learning
LMS or other tracking data
LMS or other tracking data
Accounting or other budget systems
Surveys (immediately after and delayed)
Correspondence with customers, surveys
Feedback, requests to training department

Business measures
Reduction in training time LMS statistics, ROI, surveys, comparisons of
instructor-led to e-Learning
Reduction in training costs (time, travel, etc.) Cost measures
Increased customer job satisfaction Surveys
Customer certifications Tests, tracking inside courseware
Reduced customer support costs Comparison of performance data over time, surveys
and phone calls
Increased customer job performance Interviews with sales people
Increased sales Cost measures
Reduced process time Profit measures

Kirkpatrick Evaluation Model

Donald Kirkpatrick was ask by the ASTD in the mid-1970’s to conduct a study of what evaluation techniques where being
used in industry and education. His report quickly became known at the Kirkpatrick Evaluation Model and has been used
widely for over 30 years. In one recent article, the question was asked: Is Kirkpatrick Obsolete? Alternatives for Measuring
Learning Success (http://www.ltimagazine.com/ltimagazine/article/articleDetail.jsp?id=85206). This article raised the issues
of using Voice of the Customer (VOC), Voice of the Business (VOB) {we might say Voice of the Process - VOP} and how
this all can relate in Six Sigma terminology.

E. Rice in the Instructional Technology or (Educational Technology) Masters and Doctoral Study and Reference
Deck developed the following table for the Kirkpatrick Evaluation Model:

Level Advantages Disadvantages

Level 1: Reaction. Smile sheet. Asks
for how the student feels about
training.
Level 2: Learning. Measures whether
or not a learner has actually learned.
Pre- and post-tests are examples. So
are performance observations,
interviews, self-assessments.
Level 3: Behavior. Change as a result
of learning. On-the-job observations,
self-assessments, measurement of
changes in performance.
Level 4: Results. Shows impact on
the organization.
Immediate, quick. Can answer
questions about classroom, atmosphere,
instructor performance.
Reflects what’s been taught.
Can compare results to a control group.
Gives objective measurement.
Allows for different measures.
Proves actual performance has
changed.
May justify the cost of training. Will
prove if it works.
Can generate false data, or data that has
no use or is superficial.
May be done too soon after training.
Does not measure transfer to the job.
Hard to measure. May not be accurate.
Most companies don’t want to know.
May have negative bottom line impact
if it didn’t work.
Popular in the automotive companies is to add a 5th Level: Return on Investment (ROI) Breaks down Level 4 “impact” into
two distinct areas.

Rice & Munro Training Evaluation Model

The Rice & Munro Training Evaluation Model evolved out of the Kirkpatrick Evaluation Model during the Third
Annual ISO 9001 Conference in the mid-1990’s and can be found listed in the ASQ CQE Handbook. E. Rice in the
Instructional Technology or (Educational Technology) Masters and Doctoral Study and Reference Deck describes the Model
this way:

“The Rice and Munro Evaluation Model was developed in response to the frustration experienced by many trainers
to the lack of management support for conducting more substantive evaluation, such as Kirkpatrick Model Level 3 and/or 4
or the like. The Rice and Munro Evaluation Model (RMEM) uses both the foundations of the Kirkpatrick Evaluation Model
and the need for continual auditing - using a process approach found in the ISO 9000 and ISO 14000 International Standards
for Quality Management Systems and Environmental Management Systems respectively. The integration of these evaluation
and auditing methods highlights the company as a System.

In short, the RMEM procedure includes:

1. Training Manager reviews the need for Kirkpatrick Levels 3 and 4 within the organization.
2. Training Manager meets with the Audit Manager(s) (QMS and EMS) for a review of upcoming audits in the
company’s annual audit plan.
3. Training Managers provides to the Audit Manager(s) names of individuals from the areas to be audited that have
attending training programs in the past six to eighteen months and what questions need to be asked of these
individuals related to the application of specific training course objectives and expected outcomes.
4. Audit team conducts audits (QMS or EMS) and asks identified individuals questions relating to the training
programs that they attended. These findings are included in the organizations internal audit reports.
5. Audit reports relating to the training items are shared with the Training Manager who includes the information in the
training history information for ongoing Level 3 and 4 evaluations.”

A key to good e-Learning evaluation is the fact that plans should be made during the design/development phases to
conduct the desired level of evaluation after the learner has completed the learning. Many times, learning evaluation is only
an afterthought and many e-Learning sales people and organizational managers are not well versed in what goes into a good
learning evaluation. Far too many times, we see a manager wanting only quick evaluation results immediately after the
learning has been finished and thus the usefulness of learning evaluation has been in question for many years. Good
evaluation take time and energy to conduct.

The reader may also wish to review: Resources Needed to Make Evaluation Work
(http://www.learningcircuits.org/2004/nov2004/phillips.htm) by J. Phillips, P. Pulliam Phillips, and T. Krucky Hodges. This
website is an excerpt from the new ASTD Press book, Make Training Evaluation Work, outlines the technology (both
hardware and software) and time requirements needed to implement evaluation, as well as the budgetary requirements to
make it successful. The time requirements to conduct evaluations can be seen in the follow graph:
CONCLUSION

E-Learning can mean many different things in today’s technology and be applied in many formats (see figure 2). But
selecting the methods of learning to fit the situation is still the most important point in deciding what will be useful within
your organization (reference the job aids at the end of this paper). The quality professional should use some very common
tools and techniques in deciding when to use off-the-shelf programs or when to have technology developed around a learning
initiative. These tools start with VOC, AQP, COPQ and auditing results.

Common E-Learning Traps (Rossen & Hartley 2001)

There are a number of things that can hinder the promising e-learning solution. Being aware of these traps will help your
prevent them. Some common problems include:
• Becoming enamored with the extraordinary features that accompany e-learning and seeing it more as a “gadget” than
focusing on learning the content it delivers
• Feeling intimidated and overwhelmed by e-learning
• Expecting e-learning solutions to completely replace face-to-face instruction
• Failing to approach e-learning as a collaborative and team-oriented effort
• Neglecting to simultaneously implement change management
• Assumptions of uniform operating systems, learning styles, motivation, frustration levels, time management, and
technology savvy
• Thinking too big and too fast

In recognizing that people learn in different ways in different settings (see figure 2), the manager or quality practitioner
should always be aware that when dealing with people, it is always best to follow the platinum rule – do unto others as they
would like to be done unto! Thus, the advice of Philip Crosby may sum our review up the best: “The educational needs of
each employee should be noted as part of his or her professional review… People should be encouraged to read widely and to
study what they wish - just to study.” Crosby thought that managers would do well to continually encourage their employees
to learn new things on a regular basis. With today’s technologies (from books to intranets to internets) the ability for people
to continually learn is becoming easier and easier.
REFERENCES

Boehle, S. 2005, “Simulations: The Next Generation of E-Learning”, Training Magazine, January 2005, Vol 42, No
1, pp. 22-31.

Burns, T., 2005, “E-Learning: The Future of Quality Training”, Quality Progress, February, Vol 38, No 2, pp. 50-
56.

Crosby, P.B., 1986, Running Things: Art of Making Things Happen, McGraw Hill, New York: NY, page 152.

Rice-Munro, E.J., 2003, Instructional Technology or (Educational Technology) Masters and Doctoral Study and
Reference Deck, RAM Q Universe, Inc., Reno, NV.

Rossen, E., Hartley, D., 2001, Basics of E-Learning, American Society for Training and Development – Info-Line,
Alexandria, VA, September, Issue 0109.

Rosenberg, M. J., “Redefining E-Learning”, Performance Improvement, March 2003, Vol 42, No. 3, pp 38-41.

Stolovitch, H. D., 1992, Handbook of Human Performance Technology: A Comprehensive Guide for Analyzing and
Solving Performance Problems in Organizations, Jossey-Bass Publishers, San Francisco, CA.

Additional Sources Of Information (Besides The Internet)

ASTD - American Society for Training and Development - http://www.astd.org/

ISPI - International Society for Performance Improvement - http://www.ispi.org/

Chief Learning Officer - magazine: http://www.clomedia.com

newsletter: http://www.submag.com/sub/ch?wp=wpdly
Educational Technology - http://bookstoread.com/etp/
Training Magazine - http://www.trainingmag.com/training/index.jsp
Job Aids – found in the ASTD Info-Line: Basics of E-Learning by E. Rossen and D. Hartley (individual copies of Info-line
topics can be purchased from ASTD).

Evaluation Worksheet

Following is a worksheet to help you evaluate whether e-learning is appropriate for your organization now. As every
organization is different, and constantly changing., there are no black-and-white answers. Accordingly, use your discretion
in applying this job aid to your training solutions.

• Do you have a training strategy that requires you to provide training at multiple sites or to traveling employees?
• Is your organization rapidly changing, thus requiring the need for fast and efficient acquisition of new knowledge and
skills?
• Does your organization have a commitment to self-directed learning?
• Do you have many courses, but not enough instructors with the specialized skills to teach them?
• Do you have the appropriate technology to consider implementing e-learning? If not, can you arrange to buy or lease it?
• Do you have the support of your Information Systems department?
• Do you have a change management initiative in your organization that supports e-learning?
• Do you have instructors available to facilitate e-learning that are specially trained?

If you can answer yes to most of the above questions, your organization is likely ready to move forward and take part in the
e-learning revolution sweeping the learning community. However, even if you have quite a few no answers, assess where the
weaknesses are and use this Info-line to determine what you can do to remedy them

How to Develop an E-Learning Initiative

Proving your case to the powers that be within your organization can be one of the hardest steps in developing an e-learning
program. Use the following worksheet to outline your ideas and establish a sound business case for an e-learning program.

• Background information:
• Purpose of e-learning initiative:
• Benefits for the learners:
• Benefits for the organization:
• Risks associated with move toward e-learning:
• Policies and guidelines:
• Current organizational successes:
• Strategy:
o What do we need to track, measure, and report?
o What is best taught with e-learning?
o What is best taught with more traditional methods?
o What will we accommodate? What will we change?
o What standards will we use to govern our e-learning?
o What will it take to manage this?
o How will e-learning be managed?
o How will the learner manage e-learning?
o What are the marketing/communication issues?
o Who do we have to partner with?
• Summary: