Return-on-Investment

IN THIS ARTICLE

ROI
By Amy Purcell Heres how LensCrafters, Sears, and Apple prove a clear link between training and financial results.
t was 1992 and Dave Palm at LensCrafters picked up the phone. It turned out to be the ultimate wakeup call. Palm, then training director, got the message loud and clear: Show us the return-on-investment or. A frantic regional manager had called to tell Palm that LensCrafters executives were looking to improve the bottom line and couldnt find enough evidence that training programs were providing a quantifiable return on the companys investment. Yes, the manager said, executives knew LensCrafters associates had to be trained well to craft eyewear. Yes, they knew associates were satisfied with the training. And, yes, they were aware of the qualitative evidence that performance improvements among 12,000 frontline associates could be attributed to training. But they wanted to know whether all of the bucks were giving the intended bang. So, Dave, the regional manager asked, What are you going to do about this? Click. Gulp.

20/20

Many training professionals have a lovehate relationship with ROI: They love to hate it. Its time-consuming and can be complex, and it requires identifying and using appropriate quantitative measures. Many trainers dont take evaluation to Kirkpatricks level 4, results; even fewer take it to Jack J. Phillipss level 5, ROIthe monetary value of training results exceeding the cost of training. Conducting ROI using level 5 or similar statistical equations can t help you talk the same financial language spoken by other members of the management team t increase your credibility and involvement in strategic decision making t get you more human and financial resources t measure your personal performance development based on business results. Steve Kirns, vice president, innovation and organization development for the Sears Company, knows the excuses but says train-

The Gist
Some ROI measures require a good understanding of multivariance statistics and complex modeling techniques. t A good way to track performance is to look at sales, profits, and customer satisfaction numbers and see where the correlations are. t Proving ROI to top management can be a career booster.
t

28

Training & Development, July 2000

ers need to get better at measuring ROI or risk being cut out. Says Kirns candidly, Training isnt something we do because its cool or the company thinks its cool. We do it because its a viable way to help the organization grow and make money. [We still seem to] measure our training function by the activitiesrather than the results. Thats fine up to a point, but there comes a time when you have to get down to business, and that means measuring ROI. Frankly, ROI is hard and people are busy trying to do what they do best, which is design and implement. Some of the ROI measures weve conducted at Sears have required a good understanding of multivariance statistics and complex modeling techniques. Kirns, who has been with Sears for seven years and has a Ph.D. in clinical psychology, is no stranger to statistics. But how do other learning performance professionals get down to the business of measurement? Determining the value of training within a companys overall business strategy is the first step to deciding where, when, and how to conduct ROI measurements. Then, you need to understand your organizations overall business goals and the pain associated with them. To uncover the pain, ask t What metrics and measures does the organization value? t What are the future strategies? t In which areas are key financials faltering? t What quantifiable outcomes are you looking for? Improved customer service? Increased productivity? Higher profits? Less turnover? All of those?

Training... We do it because its a viable way to help the organization grow and make money.
Working with operations upfront sets the tone early on that the intervention youll be designing will address its concerns. Once the pain is out in the open, you can partner with management to design an intervention that will help change or treat the root cause. Often, that can mean teaming up with your worst enemies number-crunching employees skeptical of your ability to link training to profitability, higher-quality output, and increased sales and satisfaction. When Palm received that wakeup call seven years ago, training at LensCrafters was decentralized into three independently operating geographic regions. With no time on his side, he turned to his top trainers for ideas and chose a lesstraveled ROI path: partnering with the operations branch of LensCrafters. All of our executives have had operations experience, says Palm. Their mindsets are focused on how to improve operations to remain the industry leader. By aligning with operations, were now seen as a key strategic partner that can deliver tangible results. Palm says that as a team, its important to t identify clear measurements that will determine whether the training intervention is successful t state what business problem needs to be solved and how training can solve it t show your partner how youll track results pre- and post-training. Key to developing your ROI metrics is to gain agreement from your customers that the measures youre designing will show whether the training is successful, says Palm. Its crucial that your customers understand and agree with you about the numbers. Otherwise, they will be looking for different information from you in the end, and the program will fail before it even gets started.

Palm decided to test a link between training and key business drivers by measuring each store location in two of LensCrafterss nine regions. The areas to be examined: waste from mistakes in quality and remakes, store performance and sales, training saturation, and the companys five specific customer satisfaction behaviors, as measured monthly by an outside firm. Palm compared the results of the control and test regions. Results showed that all stores in the test regions reduced waste, increased sales, and improved in all five customer satisfaction areas. As the ROI model matured, Palm also proved region by region that a correlation existed between the highest level of training saturation and the best or most improved business results. That got managements attention. Now, LensCrafterss training department has 10 times the resources it had seven years ago, and the training budget has more than doubled, with the company earmarking $10 million a year for program development and executionmore capital investiture in training than by any other optical retail competitor. Moreover, because LensCrafterss training department is considered a true business partner, it received budgetary funds to develop a multimedia-based system, from which the savings have been reinvested in management and leadership programs.

Determine which programs

When Holly Burkett started at Apple Computers Sacramento, California operations office, the facility was concerned about the quality and consistency of the on-the-job training of its 1,700 full-time and temporary assemblers. Burkett, senior training and development consultant and a Performance Resources Organization Certified ROI practitioner, wanted to take the facilitys evaluation efforts to new levels. After finding that the organizations pain was not only in the area of on-the-job training, but also in post-process audit failures caused when an employee would perform a task in error, Burkett proposed a train-the-trainer intervention to measure change in workers behavior and to measure the companys ROI on the training. With the help of senior management, Burkett established evaluation targets for

Partner up
Identifying the root cause of organizational pain around some areas is easy when you partner with top management to analyze financial results, customer satisfaction ratings, and key business drivers. The organization may already capture many of those numbers, which just have to be collected and collated meaningfully by you. We found it was easier to uncover the pain when we talked with our operations teams, says Palm. If you look at the operations side of your business, you may have a better time finding the numbers you need to create ROI calculations.
30
Training & Development, July 2000

all of the facilitys training programs so there would be no question about what they were measuring for. She then classified each program into its evaluation and measurement levels, determining that all of the programs delivered at the facility would receive a level-3 evaluation, with 20 percent also receiving level 4 and 10 percent receiving level 5. Why not measure all programs at level 5? Burkett contends that only programs identified as business-critical those that address a high-risk business issue or have the greatest impact on the bottom lineshould receive level-5 evaluation. In the real world of ROI, not all programs can be linked to key business measures, says Burkett. Its timeintensive work, and there are certain interventions for which the results cant be linked to the bottom line in a valid, credible way because you cant isolate the training from other variables. Kirns agrees and cites an example: When Sears decided to sell DVD players, employees needed product knowledge and training. But there wasnt a valid reason to spend time and money conducting ROI for that training because there wasnt any pain associated with selling DVD players.

That can mean teaming up with your worst enemiesnumber-crunching employees skeptical of your ability to link training to profitability.
breaking down tasks will improve productivity or efficiency). Participants assigned a confidence percentage to their estimates, with the average being 70 percent. Burkett multiplied that by the average overall monetary value of the cost-benefit each participant selected. For example, participants estimated a total overall monthly cost benefit of $336,000, with associated business improvements, and showed an average 70 percent confidence level with that estimate. Seventy percent multiplied by $336,000 is $235,200. That was divided by 20 respondents, resulting in an average estimated cost benefit of $11,760 to a trained line over a 30-day period. When the pilot was completed, Burkett was able to show a marked decrease in process failures in the test group over the 60-day period. The untrained control groups experienced 2,000 more minutes of downtime than the trained groups combinedwhich, for Apple, translated into more products built and shipped to its customers. LensCrafters tests likewise. For instance, it tested its new multimediabased training system before rolling it out to 861 stores. The test outcomes: t Training time was reduced by 40 percent, saving the company $1.2 million annually. t Retention improved 10 percent. t Two customer satisfaction measuresAre associates knowledgeable? and associates ability to serverose 10 percent. t Associate satisfaction saw positive gains, and turnover was reduceda savings of $2.3 million per year. Recently, Sears wanted to test the way it merchandised tools and hardware. With 17 stores in the test and an equal number in the control group, employees

in the test group received a two-day workshop on the new Tool Territory merchandise setup. Sears saw a 20 to 25 percent increase in sales in the test stores. The Tool Territory concept is now rolling out to 200 stores. Apple, LensCrafters, and Sears say that a pilot test environment makes it easier to isolate training from other variables. Which begs the question: How do you isolate?

Isolate
Performance improvements can be linked to training, but nontraining factors may also contribute to improvement. Isolating is a crucial step, yet it remains one of ROIs stickier wickets. Though it was evident that training played a huge role in Searss ability to increase tool sales, Kirns maintains that its often difficult to isolate training when measuring ROI or performance results unless youre willing to devise controlled experiments. If thats the case, could the higher sales of Searss Tool Territory have occurred without the training? Doubtful, says Kirns. But training is just one aspect of improving performance and business results. The key for us is that our company sees training as an integral part of the business, not as a department sitting on the sidelines. Were at the table every time the company is talking about business strategy, and were contributing ideas and information to improve the bottom line. Palm agrees, saying that top management wont just accept your ROI calculations. Theyll want to know what factors went into them and what other factors may affect them. Suggests Palm, If youre unsure how to develop an ROI calculation, consider checking with other departments such as market research or finance or those that administer TQM. Those departments are accustomed to using statistical models and can keep a trainer from having to reinvent the ROI wheel.

Start with a pilot

Regarding ROI, the use of test-control groups adds authenticity to your analysis. The test group receives the training; the control group doesnt. For validity, use standard survey practicessuch as random selection and similar demographics and environments. Burkett devised two separately trained groups of 27 employees each (test) and two untrained groups with an equal number of participants (control). She also had a pretraining history of what was happening on the production line in each area she was measuring: productivity, quality, and labor efficiency. Last, she evaluated business impact by comparing performance of post-process audit related to downtime between the control and test groups for 60 days following the training. Using a template from Phillipss level-5 model, Burkett asked each participant in the test group to estimate the effect of his or her behavior change on a particular business measure (such as,

Calculate
ROI can be simple or highly complex, depending on what youre measuring and what other variables may affect that measurement. The key is to start small and get it right the first time. That way, you establish your credibility. According to Palm, it goes something
Training & Development, July 2000 31

like this: t Find out where the pain is. t Determine how much money that pain is causing the company to lose. t Create a pilot test. t Develop a cost-benefit ratio of the solution, comparing how much money was earned by implementing the training with how much money was spent. Then, as you get more comfortable with ROI, you can branch into larger efforts and more complex statistical models. Soft data can also be converted to monetary values and compared with the cost of the training by following Phillipss five-step conversion method. (See How Much Is the Training Worth? Training & Development, April 1996.) t Focus on a single unit of output, quality, or time. If its soft data, focus on one case of employee turnover or a one-point change of the customer service index. t Place a value on the unit that you have identified. t Determine the performance change after factoring out other potential influences on the training results. t Obtain an annual amount. The industry standard for an annual performance change is equal to the total change in

Only programs identified as business-criticalthose that address a high-risk business issue or have the greatest impact on the bottom lineshould receive level-5 evaluation.
performance data during one year. Multiply the annual performance change by the unit value. t Compare the product of your equation to the cost of the training using this formula: ROI = net annual value of improvement program cost. There are other benefits than dollarsand-cents savings. For one, Burkett says that conducting ROI measures has been a career booster. Shes one of the only facility trainers to link business performance with her programs. For me, its more empowering to know that our departments work has a
t

direct impact on performance, productivity, or sales than it is to know that people enjoy the training program, says Burkett. Once I came to terms with the fact that I would get more support and have more influence over our stakeholders if I could use the hard data they value to my advantage, I gained more confidence and competence. Palm agrees wholeheartedly, adding: Our successes at LensCrafters have been enough to show top management that an investment in training is a good investment in the future of the business. We dont expect to be the end-all in the company, just an important part of the bottom line. If employees are applying the training, the end result should be better performance. And theres no better way to track performance than to look at sales, profits, and customer satisfaction numbers and see where the correlations are. Bottom line: Were not here to train numbers; were here to get business results. Its about helping our company be successful. t
Amy Purcell is communications super-

visor at LensCrafters; apurcell@ lenscrafters.com.

32

Training & Development, July 2000