Learning Metrics That Matter

Learning
metrics that
matter
You can't ignore
ROI any more

Brought to you by

Learning metrics that matter

Contents
The Metrics Challenge

Page 3

What are the metrics that matter?

Page 6

10 tips on how to develop learning metrics that matter

Page 9

Case study: A simple approach to measuring ROI

Page 10

About Bray Leino

Page 11

About TrainingZone

Page 12

Learning Metrics That Matter

The metrics challenge

Measuring the impact of learning is a major challenge for L&D and
something it has historically struggled with. In this whitepaper we
will explain why it is crucial that L&D overcomes this challenge and
starts measuring the impact of learning. The result will be some really
interesting data and insights that will inform L&D, boost its credibility
with the business and drive business success.
For years L&D professionals have protested that its too tricky to prove the direct impact of
most learning interventions. That wont wash any more. L&D simply must demonstrate how
it is adding value to the business and what the impact is of learning on the bottom line.
Firstly, L&D has to move away from the old way of measuring learning namely, how many
people turn up for and complete training and what ratings learners give courses.
Measuring learning is now all about the impact of learning and data that demonstrates the
value it adds. What does learning deliver in your organisation? If L&D wants the board to
sign off the investment in a piece of training then it will need to show the board why it cant
afford to say no. Highlight what the training will achieve, demonstrate how the business
will benefit and where the value will be added. Boards want clear, evidence-based data that
spells out exactly why learning needs to happen and how the business will benefit.
Learning metrics have to focus on outputs. Concentrate on the desired outputs when
designing learning interventions and keep assessing those outputs throughout the process
to see if they are being met.
The benchmarking organisation Towards Maturity has spent a lot of time talking about the
importance of L&D having outputs at its core, rather than inputs. Its research shows that
organisations that focus on outputs are reaping the benefits. Towards Maturitys 2014-15

Learning Metrics That Matter

benchmark study, Modernising Learning, Delivering Results, found that organisations that
focus on critical outputs rather than inputs are the most aligned to the business and the
more likely to succeed. They are:

13x more likely to report increased revenue

9x more likely to report increased productivity
5x more likely to report improved customer satisfaction
Historically, L&D data has been about inputs that bums on seats and training hours
data mentioned earlier. This kind of data tells L&D and the business very little about how
effective learning has been and what impact it has had on performance. So an individual
attended and liked a training module, but has it actually improved their performance at
work? Has it helped them achieve what needed to be achieved? Has it added value to the
bottom line?
When this type of inputs data is the only data that is collected,
L&D doesnt really know how effective learning has been.
Hence, L&D does not have a good track record when it comes to
collecting data. Nor does it have a good track record of sharing
that data in terms that the business either understands or needs.

"There has been a lot

of conflicting advice
on how data can and
should be collected."

In L&Ds defence, there has been a lot of conflicting advice on

how data can and should be collected, which assessment models
to use or not. There are those who swear by the Kirkpatrick
model, for example, and there are those who think it offers little real value. One of the
criticisms levelled at the Kirkpatrick approach is that although there are four levels of
evaluation, a lot of the evaluation only extends as far as level one what a learner felt and
thought about a piece of training. In order to generate any real insight to the success of
learning, all four levels should be evaluated. The other three levels look at:

Learning how knowledge or capability has increased

Behaviour - the extent to which behaviour and capability has improved
Results what benefits the learning has brought to the individual and the business
These levels are the ones that offer real insights.
Not only have learning metrics changed, but the whole context of learning has changed,
largely driven by the technology revolution. Learners are increasingly in control of their own
learning. Technology enables them to access learning so easily and quickly that they can
learn when they want, how they want and what they want. If L&D wants to capture, analyse
and share data on how learners are learning as a whole, then it cannot focus solely on those
training elements provided by L&D. It needs to look at and be involved in the bigger picture.
Also, technology has provided us with the tools and systems to track, analyse and
communicate metrics of the impact of learning in organisations. They exist. L&D must use
them and use them properly.
In the new learning landscape, learning has become much more targeted, strategic and longterm. L&D needs to foster a learning culture, where individuals are encouraged to learn all

Learning Metrics That Matter

the time. Towards Maturitys research shows that a learning culture pays dividends the
top learning companies in its benchmarking research invariably support continuous, longterm learning in the workplace.
As with anything, context is critical. All organisations are different, learning environments
are different and the success criteria will be different. There is no one-size-fits-all
answer. L&D practitioners have to work out how best to evaluate the impact of learning
so as to obtain the most meaningful results. Whatever evaluation model or models
L&D practitioners choose to use, the number one criteria is that the results have to be
meaningful. L&D has to measure the right things and for the right reasons.

Learning Metrics That Matter

What are the metrics

that matter?
Learning metrics really need to focus on two key areas: behaviour
change and performance impact. What does learning need to achieve?
It often requires behaviour change say performing a task in a new
way or managing employees more effectively, for example and always
requires a performance impact.
This sounds simple and obvious. What is less straightforward for L&D professionals
is how to measure and demonstrate how learning has led to behaviour change and
performance impact. How do you collect, analyse and present data that demonstrates
that a manager has become a better manager? That is where L&D has always said the
ROI cannot be proved. That mind-set needs to be overcome.
To begin with, it is essential that L&D really understands the starting point. When a
senior manager knocks at the door with a learning need, dont jump straight in with a
solution. Find out what the problem or what the learning need is. Gathering baseline
metrics is really important. If L&D can get the metrics right at the diagnostic stage, it
is so much more likely that the learning will achieve the desired results, and you will
be better equipped to measure its impact. Too often L&D has focused on solutions
and giving the business what it asks for without making sure it properly understands
the problem or challenge that the learning is supposed to address. Always ask and
understand what the learning needs to achieve. Then think about how data can help
further that understanding. The more evidence L&D has from the outset, the better
informed it is to make the right decisions and to measure the data as time progresses.
The data required is increasingly multi-dimensional.
What metrics might L&D need to collate? It could be typical HR data such as attrition rates,
absence rates and engagement levels. However, HR shouldnt just focus on traditional HR
data business metrics are hugely important too. The learning is supposed to improve
business metrics, right? Then business metrics need to be part of L&D metrics.

Learning Metrics That Matter

Thats why L&D really needs to be a business partner. L&D needs to be regularly communicating
with the business at all levels, from the chief executive and senior managers through to line managers
and the general workforce. Know where behaviour change is needed and where performance
improvement is needed. Some people call this action inquiry constantly finding out where change is
happening, where change needs to happen and where the gaps are. If continuous learning is to be at
heart of the business, then L&D needs to be at the heart of the business.
Research from the Chartered Institute of Personnel and
Development shows that L&D professionals that provide learning
that focuses on long-term, sustainable business growth are
providing the learning that individuals and organisations need most.
One CIPD report, The Value of learning, A new model of value and
evaluation, found that organisations that encourage continuous
learning also demonstrate strong alignment between L&D and
strategic business needs.
Continuous learning is of course tied into the 70:20:10 learning
framework. As we all know, 70:20:10 works on the premise that
roughly 70% of learning happens on the job, 20% comes from social
learning with colleagues and 10% is from formal learning.

"Organisations
practising the
70:20:10 model
are enjoying
significant
benefits."

L&D may say How can we evaluate and prove that 70:20:10 works if the majority of the
learning happens informally? Towards Maturitys research shows that organisations
practising the 70:20:10 model are enjoying significant benefits. According to a report,
70+20+10=100: The Evidence Behind the Numbers, produced by Towards Maturity in
conjunction with the global learning expert Charles Jennings, such organisations are at least:

Four times more likely to respond faster to business change (30% versus 7%)
Three times more likely to report improvements in staff motivation (27% versus 8%)
as likely to report improvements in customer satisfaction scores
Twice
(42% versus 18%)
When done well, blended learning falls into the 70:20:10 camp. However, blended
learning has come in for some criticism recently, with detractors saying that unless L&D
really understands what blended learning means and how to achieve it, then it ends up
being traditional training with some e-learning thrown in at the end of it. L&D needs to
ensure this doesnt happen because a well thought out and properly designed blended
learning approach can yield significant results. For one thing, effective blended learning
- providing a mix of learning styles and interventions helps build a company culture of
continuous learning. Learners can access learning materials anytime and any place. That
means accessing learning more quickly which in turn helps apply it more quickly. Thats
when the learning has a greater and faster impact.
To find out what learning is working and what needs improving, L&D needs to keep
collecting and analysing data and keep talking to people. L&D should be measuring
areas such as engagement levels. Tools that measure culture and values can be a very
reliable way of measuring impact. Focus groups, questionnaires and regular, in-depth
conversations with managers about the impact of learning are all important ways that
L&D can and should evaluate the impact of learning.

Learning Metrics That Matter

The role of managers

Managers have a critical role to play in the creation
of good learning metrics. After all, they are the ones
that really benefit from a strong learning culture.
They should be flagging up performance issues and
those areas that need improvement with L&D in the
first place. They should also be supporting, enabling
and reinforcing performance and behaviour change.
L&D need to have managers on board if they are
to get the metrics required and if learners are to
consolidate their learning in their roles.
However, L&D may have some work to do both in
terms of getting managers on board and ensuring
managers are able to have the honest, constructive
and meaningful conversations with employees
about learning, behaviour change and performance.
A recent survey by the US organisation Interact
found that 69% of the 616 managers polled often
felt uncomfortable communicating with employees.
Over a third (37%) said they were uncomfortable
giving direct feedback about an employees performance if they thought the employee
would respond negatively.
Yet, the ability to manage, communicate and coach employees is critical to being a good
manager, particularly given that most learning happens on the job. L&D needs to ensure
that managers have the capabilities and aptitudes to manage their employees well, that
they take learning seriously and are able to provide the kind of learning opportunities,
insights and coaching that is needed.
Managers also form an important part in the evaluation process, giving their feedback
about performance improvement. If behaviour change was needed, do they think it
has it happened? Where has performance improved? Where has it not? What are the
obstacles to change? Managers need to assist L&D in the provision of learning metrics.
Telling success stories
As we know, organisations now want clear, evidence-based data on how learning is having an impact.
L&D must share impact data with the business and it must be in business terms. Highlight all the
outcomes from the data some of the insights might be very surprising. L&D needs to be a lot savvier
about giving business leaders what they want. Identify who needs what data, how they want to
receive it and then give it to them.
Everyone loves success stories, so give the business success stories. L&D really needs to show
how learning is having an impact so use success stories to do that. Gather the necessary data and
anecdotal evidence by having good conversations with people in the business, through running
focus groups and questionnaires. Then pull all the data together, analyse it and share it with the
business. Shout about successes as it will lead to further successes.

Learning Metrics That Matter

Knowing that learning is making a difference will also engage learners to keep on learning. By showing
everyone in the business from entry level to CEO that learning is having an impact, it will encourage
more learning. This creates a virtuous circle of deliver measure share deliver and so on.
Conclusion
The new learning landscape is here and will keep evolving. L&D needs to be part of this
landscape and keep evolving with it. New learning metrics are here too and they will
keep evolving. L&D needs to evolve with them. It is time for L&D to stop feeling daunted
by data and start harnessing it. These new learning metrics are very exciting and offer
real benefit to L&D and the business. Sticking with the old way of measuring learning is
not an option L&D has to keep up with the times.
We would like to thank Karen Partridge for her input in this whitepaper.

10 tips on how to develop learning metrics that matter

1. Focus on outputs not inputs.
Collect and analyse the data that will give
meaningful insights into why learning is
necessary, where its working, where its
not and why. Always look for the behaviour
change and performance impact.

2. Know why learning needs

to happen.
And what it needs to achieve.

3. Know the starting point.

Gather metrics from the outset it helps
ensure the learning has the right focus and
provides data to refer back to and measure
the impact of learning.

4. Be a business partner.
Keep talking to the business so that you
know where the learning needs are and can
help identify gaps.

5. Be business-minded.
Learning metrics need to be aligned to
business metrics so that learning meets
business needs.

6. Use technology.
Gather, analyse and disseminate data.

7. Offer choice.
Use a variety of learning methods and
evaluation methods.

8. Continuous learning.
Foster a culture of continuous, long term
learning.

9. Get managers on board.

Ensure they take learning seriously, that
they have the capabilities to manage tricky
conversations and help achieve behaviour
change.

10. Share success stories.

Success breeds success. Spread the word
and increase buy in.

Learning Metrics That Matter

10

Case study: A simple approach

to measuring ROI
Overview
Our Foundation Leadership Programme introduces the concept of return on investment
early on. It helps delegates understand how they will be required to report on their own
ROI. Just as importantly, it highlights the concept that time is money, and that they could be
wasting money in many different ways without realising it.
We offer a very simple but effective process to help delegates identify their ROI, improve it
and then be able to prove it.
Delegates are encouraged to keep a learning journal throughout the programme to help
them critique and report on their final ROI. This also embeds all the learning, demonstrates
how it is having an impact and how their new skills are benefiting themselves, individuals
and their organisation.
Heres an example of how an L&D delegate might be able to improve, track and prove the
ROI of a given intervention, using this process:

Select something you have learnt:

How to manage difficult conversations.
Work out what the current situation is costing you:
A manager is spending far too much time preparing for a difficult conversation. They are
stressed at the prospect of the conversation and it is impacting on their effectiveness at
work. Over a period of a week, four hours were lost, preparing and worrying. Meanwhile,
the employee concerned is causing other problems in the team, problems that the manager
then has to deal with.
Plan how you are going to apply the learning. Note what changes you expect to see:
The manager decided to use the planning tools, questioning and listening techniques
covered in the programme. The manager used the journal to track progress and to reflect on
and review how the plan might need adjusting.
Apply the learning, following your plan:
Because the manager felt properly prepared, a lot of the stress fell away. The manager
stopped worrying and was able to conduct the meeting effectively and kept refining their
performance over a couple of meetings. In the end, the manager felt completely comfortable
with the conversation.
What difference has it made? This is your ROI:
The process saved the manager roughly three hours in one week.

Learning Metrics That Matter

11

About
We are passionate about change and creating robust, modern learning
cultures for your people. Well always go over and above to get the job
done.
Whatever it takes.
Understanding your business and working closely with you to deliver full circle solutions is
what we do. Our passion is helping you to develop your people to deliver results, regardless
if their role is entry-level or your CEO.
The world is constantly changing. And in business, to stand still is to fall behind. For over 20
years weve been helping our clients and their brands thrive in an ever-changing world by
embedding skills change, behaviour change and knowledge change every time.
Our four current focus areas are:

Encouraging learner engagement, ultimately improving ROI

Collating and re-engineering your learning content into a modern blend that works
Addressing the disparate needs of your people
Developing individuals for their journey to the top
We embed modern learning cultures, transforming your business through blended learning,
and shifting the responsibility of development to your people in a way that engages and
excites them, through practised pull learning strategies.
Not only is this model results-driven, but as the number one provider of blended learning,
we know that this means a reduction in learning and development costs.

For more information, visit

www.brayleinolearning.co.uk

Learning Metrics That Matter

12

About

With over 100,000 members, TrainingZone.co.uk is the largest online

community for UK learning and development professionals.
Profiled members can view and download reports, toolkits, guides and research papers that
feature the latest insight and thinking on all the cutting edge issues within L&D and HR.
Elsewhere onsite TrainingZone also publishes in-depth features written by industry experts,
as well as hosting highly interactive community areas where peers discuss and dissect the
issues that really matter.
Join the community at

www.trainingzone.co.uk