The dangers of Likert scale data

February 18, 2014advanced stuffinterval data, Likert, ordinal data

Imagine that you want to compare two products A and B and you ask the opinions of 100 users
via a survey. The table below shows a summary of the survey and the responses. The numbers
under product A and product B show the number of people who gave each of the responses on
the left-hand side.

This is known as a Likert scale and this post will give some thoughts on how to analyse these
data.

The first thing that is worth mentioning is that there is a simple form of analysis that is relatively
uncontentious. This is to say that 60% of people were very satisfied or quite satisfied with
product A whereas only 45% of people were similarly very satisfied or quite satisfied with
product B. On the one hand this is simple. However, can we use this analysis to say that product
A is better than product B? Note one problem straight away, which is that 20% of people are
very dissatisfied or quite dissatisfied with product A whereas only 15% of people were similarly
very dissatisfied or quite dissatisfied with product B. It seems that product A tends to polarise
opinion and it is not clear what conclusions can be drawn.

However, quite often we assign numbers to the categories (such as 5 = very satisfied, 4 = quite
satisfied, 3 = neutral, 2 = quite dissatisfied, and 1 = very dissatisfied) and when this is done we
can produce a number for each participant’s response; we can then average this to produce the
mean values shown in the figure above. According to this we can say that on average the
response to product A is 3.6 and to product B is 3.5. Can we now use these numbers to make the
following two statements? (1) that product A is better than product B (since 3.6 is bigger than
3.5) and that (2) both products A and B are well received by the participants (since 3.6 and 3.5
are both bigger than 3). What I want to do in this post is discuss the validity of these statements
by considering several aspects of Likert scales.

Is it valid to average the numbers?

There is a long-running dispute about whether it is valid to average the scores to produce the
mean values as in the table above. To explore this we need to introduce two types of data. The
first type are called ordinal data. This is the order in which things are. The Likert scale presented
in the table above strictly produces ordinal or rank data. Imagine that three people, Alan, Brian
and Clive run a race in which Alan wins, Brian is second, and Clive is third. Knowing the order
in which they finished is fine, but it doesn’t tell us whether Alan finished well ahead of the other
two or whether, for example, Alan and Brian were involved in a close finish with Clive a long
way behind. If, however, we know how many seconds they took to complete the race (Alan = 40
seconds, Brian = 41 seconds, and Clive = 52 seconds) we now know much more information
about the race. It turned out that Clive was a long way behind the other two. The race times, in
seconds, are called interval data. With interval data the differences between the numbers are
meaningful whereas with ordinal (rank) data they are not.

The problem with a Likert scale is that the scale [of very satisfied, quite satisfied, neutral, quite
dissatisfied, very dissatisfied, for example] produces ordinal data. We know that very satisfied is
better than quite satisfied and quite satisfied is better than neutral, but is the difference between
very satisfied and quite satisfied the same as the difference between quite satisfied and neutral?
Why am I worrying about this? Because when we assign numbers to the scale (the 1-5 numbers)
and then average the responses we are implicitly making the assumption that the scale items are
evenly spaced. We are treating the ordinal data as interval data. How can we be sure that the
participants treated the scale in this way? Would it have made a difference if we had used
satisfied and dissatisfied instead of quite satisfied and quite dissatisfied respectively? So it would
seem that is wrong to calculate means from Likert scales. If you click here you will see a post
from a PhD student (Achilleas Kostoulas) at the University of Manchester who states
categorically that it is wrong to compute means from Likert scale data. I choose this example
because it is simply and elegantly explained not because I necessarily agree entirely with his
view. It is also worth reading the article by Elaine Allen and Christopher Seaman in Quality
Progress (2007) who also take the view that Likert scale data should not be treated as interval
data. Interestingly they also suggest some other techniques that don’t suffer from the ‘ordinal-
data’ problem; for example, using slider bars to get a response on a continuous scale. However,
before you give up detailed analyses of Likert scale data I would urge you to read the paper by
Susan Jamieson called Likert scales: how to (ab)use them in Medical Education (2004: 38, 1212-
1218). Although Susan is also broadly speaking against treating Likert scale data as interval data
she does present the other side of the argument. In another paper, in Advances in Health Sciences
Education, Norman (2010, 15 (5), 625-632) argues that the concerns about Likert scales are not
serious and we should happily use means and other parametric statistics.

How much bigger do two averages need to be for an effect?

In the table at the start of this article product A and B receive scores of 3.6 and 3.5 respectively.
The paragraphs above explain that calculating these means may not be valid. However, assuming
that we do calculate means in this way, how different would the mean scores for product A and
B need to be for us to conclude that A was better than B? I have come across students (normally
in vivas) who would simply state that A is better than B because 3.6 > 3.5. To those students I
then would say, would you still take that view if instead of 3.6 and 3.5 it was 3.51 and 3.5? What
if it is 3.50001 and 3.5? Would they still maintain that A is better than B? It is clear that we need
to consider variance and noise and carry out a proper statistical test to conclude whether 3.6 is
significantly greater than 3.5. The test is called a student t-test and anyone can be taught to
perform one using Microsoft Excel in a matter of minutes. In the example at the start of this
article it turns out that there is no statistically significant difference. We cannot conclude that
product A is received better than product B.

However, can we conclude that both products are received favourably? Again, we need a
statistical test. It turns out that in this case, both 3.6 and 3.5 are statistically greater than 3 and we
can at least conclude that products A and B are received favourably. However, there is the caveat
that this assumes that we can treat the Likert scale data as interval data in the first place.

Other considerations
An interesting question is whether we should use 5-point scales at all. Would we get different
results if we used a 7-, 9- or 11-point scale? I have found one website that suggests that a 7-point
scale is better than a 5-point scale but not by much. A paper by Dawes in International Journal
of Market Research (2008: 55 (1)) looked at 5-, 7- and 10-point scales and concluded that the
results from a 10-point scale would be different from a 5- or 7-point scale (after suitable
normalisation).

Although odd-number scales (with a neutral point) are almost always used. A paper by Garland
(Marketing Bulletin, 1991: 2, 66-70) suggest that using a four-point scale (and removing the
neutral point) might remove the social desirabiity bias that comes from respondents wanting to
please the interviewer. I am not sure what current thinking is on this matter though and I would
normally use odd-number scales.

I am not providing any definitive views on these points but rather raising awareness of issues. If
you want to use a Likert scale then these are issues you need to familiarise yourself with.

My view

I will confess to having treated Likert scale data as interval data and carrying out parametric
statistics (these are statistics that use statistical parameters such as standard deviations).
However, deep down I know it is wrong. I am coming to the view that the best thing is not to use
a Likert scale at all. I think people often use this sort of scale because it seems simple. There are
ways to statistically analyse data like these and I would refer readers to categorical judgement
which is a well-used psychophysical technique. My colleague Ronnier Luo at Leeds University
has used this technique extensively for decades. However, it is far from simple to analyse the
results. I think there are better ways of obtaining information. I think use sliders bars and
allowing users to indicate using the slider bar their view between two extremes (e.g. between
very satisfied and very dissatisfied) is probably better and I will encourage my students to use
this technique in the future.