See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/257036314

Adaptation of the Rating Scale Mental Effort

(RSME) for use in Indonesia

Article in International Journal of Industrial Ergonomics · January 2013

DOI: 10.1016/j.ergon.2012.11.003

CITATIONS READS

4 954

3 authors:

Ari Widyanti Addie Johnson

Bandung Institute of Technology University of Groningen
6 PUBLICATIONS 19 CITATIONS 70 PUBLICATIONS 1,270 CITATIONS

SEE PROFILE SEE PROFILE

Dick de Waard
University of Groningen
115 PUBLICATIONS 3,115 CITATIONS

SEE PROFILE

All in-text references underlined in blue are linked to publications on ResearchGate, Available from: Addie Johnson
letting you access and read them immediately. Retrieved on: 16 November 2016
 International Journal of Industrial Ergonomics 43 (2013) 70e76

Contents lists available at SciVerse ScienceDirect

International Journal of Industrial Ergonomics

journal homepage: www.elsevier.com/locate/ergon

Adaptation of the Rating Scale Mental Effort (RSME) for use in Indonesia
Ari Widyanti a, b, *, Addie Johnson a, Dick de Waard a
a
Department of Psychology, University of Groningen, Grote Kruistraat 2/1, 9712 TS Groningen, The Netherlands
b
Department of Industrial Engineering, Bandung Institute of Technology (ITB), Bandung, Indonesia

a r t i c l e i n f o a b s t r a c t

Article history: The Rating Scale Mental Effort (RSME) is a unidimensional instrument used to measure subjective mental
Received 7 April 2012 workload. The RSME consists of a line with a length of 150 mm marked with nine anchor points, each
Received in revised form accompanied by a descriptive label indicating a degree of effort. The RSME has been widely used in
11 October 2012
Western countries (e.g., in Europe and North America), but, when translated to Indonesian, has been
Accepted 6 November 2012
Available online 7 December 2012
shown to be relatively insensitive to changes in mental effort among Indonesians, raising the question of
whether the insensitivity is related to national culture or to the translation of the scale. To investigate
whether the relative insensitivity of the RSME might lie on the translation of the instrument, a new,
Keywords:
RSME
seven-anchor point, version of the RSME was created to better reflect how the anchor points are
Mental workload interpreted by Indonesians. The new version of the RSME was compared with the existing, nine-anchor
Subjective measures point version of the instrument in an experiment in which 100 participants performed a search task of
Self-reports different levels of difficulty. Half of the participants used the original RSME and half the adapted
Culture instrument. The adapted instrument appeared to be more sensitive to changes in load than the original
RSME. However, a comparison with Dutch participants who performed the same task suggests that
culture, and not only properties of the scale used to measure mental effort, influences the measurement
of subjective mental workload.
Relevance to industry: As industrialization increases in developing countries, so does the need for low-
cost, easy to implement, and valid mental-workload instruments. We adapted the Rating Scale Mental
Effort (RSME) for use by Indonesians. The adapted RSME shows improved sensitivity, but cultural
differences in reporting load are still apparent.
Ó 2012 Elsevier B.V. All rights reserved.

1. Introduction
An increasing emphasis on safety, health, and comfort in the
workplace has made the evaluation of operator mental workload in
the workplace a key issue (DiDomenico and Nussbaum, 2011).
Mental workloadddefined as the difference between the cognitive
resources demanded by the task and the cognitive resources
available to and assigned by the worker (Gopher and Donchin,
1986)dis often measured with subjective, self-report measures,
either in isolation or together with performance measures (e.g.,
reaction time), or physiological measures such as heart-rate (Cain,
2007; O’Donnell and Eggemeier, 1986), eye movement data (Di
Stasi et al., 2011), optical brain measures (Ayaz et al., 2012) or
combined physiological measures (Ryu and Myung, 2005).
Subjective measures are popular because of their ease of use and
* Corresponding author. Department of Psychology, University of Groningen,
Grote Kruistraat 2/1, 9712 TS Groningen, The Netherlands. Tel.: þ31 50 363 3845;
fax: þ31 50 363 6304.
E-mail address: a.widyanti@rug.nl (A. Widyanti).
low cost, and because their use is relatively unobtrusive in work
situations (De Waard, 1996; O’Donnell and Eggemeier, 1986).
Although some subjective measures of mental workload have
been developed recently (e.g., Mental Workload Index; Pretorius
and Cilliers, 2007), the most frequently used subjective measures
of mental workload are The NASA-Task Load Index (NASA-TLX; Hart
and Staveland, 1988), the Subjective Workload Assessment Tech-
nique (SWAT; Reid and Nygren, 1988), and the Rating Scale Mental
Effort (RSME; Zijlstra and Van Doorn, 1985; Zijlstra, 1993; for
a review see Rubio et al., 2004). Both the NASA-TLX and the SWAT
are multidimensional scales, whereas the RSME is a unidimensional
scale. The SWAT, for example, contains the scales time load, mental
effort load, and psychological stress and takes about 45 min to
complete, not including scoring (European Organization for the
Safety of Air Navigation, 2003). The NASA-TLX includes six scales
(mental demand, physical demand, temporal demand, perfor-
mance, effort, and frustration level) and takes about 10 min of the
operator’s time to complete. The RSME, which measures only
“mental effort”, can be completed in less than a minute. An
important question is whether multidimensional rating scales

0169-8141/$ e see front matter Ó 2012 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.ergon.2012.11.003
 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76
provide benefits as compared to a simple, unidimensional scale.
One reason to prefer multidimensional rating scales is their diag-
nosticity. That is, because different aspects of load are rated, it is
possible to determine, to some extent, which factors contribute to
workload (Dey and Mann, 2010; Rubio et al., 2004; Wierwille and
Eggemeier, 1993). However, it has been argued that unidimen-
sional rating scales can be as useful and as sensitive to changes in
mental workload as are multidimensional rating scales in many
conditions (Veltman and Gaillard, 1996), and, in fact, the RSME has
repeatedly been shown to be sensitive to changes in mental
workload in both laboratory (e.g., Mulder et al., 2009; Wilson et al.,
2006) and applied (e.g., De Waard, 1996; Lin and Cai, 2009) settings.
The RSME consists of a 150-point, 15-cm long vertical line
marked at 1-cm intervals, and containing nine anchor points with
descriptive labels ranging from “absolutely no effort” (close to the
0 point on the 0e150 point scale), through “rather much effort”
(about 57 on the scale), to “extreme effort” (about 112 on the scale).
Responses are made by marking the line at the point corresponding
to the amount of mental effort taken to complete a task. The
development of the RSME was based on magnitude estimates of
effort made by students (n ¼ 39) and bus drivers (n ¼ 25) in the city
of Groningen, The Netherlands. Of the 17 labels that were originally
rated, nine labels, selected to mark approximately equal intervals
on the 150-point scale, were selected for inclusion in the final
version of the scale (see Van Doorn and Zijlstra, 1988; Zijlstra and
Van Doorn, 1985, for details). In previous work, we (Johnson and
Widyanti, 2011) reported how the RSME was translated to Indo-
nesian and used to assess mental workload along with an Indo-
nesian translation of the NASA-TLX. The major finding in that study
was that the range of ratings of subjective measures of mental
workload imposed by a demanding search task (as measured by the
RSME and the NASA-TLX) was more restricted among Indonesian
participants than among the Dutch participants that they were
compared to. The range restriction in the Indonesian as compared
to the Dutch ratings is of concern because sensitivitydthe capa-
bility to detect changes in the levels of workload imposed by task
performancedis an essential property of measures of mental
workload (O’Donnell and Eggemeier, 1986). Range restriction in
scale use thus limits the effectiveness of the instruments.
We (Johnson and Widyanti, 2011) interpreted the differences we
found in how the RSME and NASA-TLX were used by Indonesian vs.
Dutch participants in terms of culture. For example, because
subjective mental workload depends on the ability and willingness
of participants to report their mental state, differences in how rating
scales are used may be influenced by the fact that the expression of
thoughts, preferences, and feelings is not encouraged in Eastern
cultures, such as that of Indonesia, whereas such expression is
commonly valued in Western cultures (i.e., in cultures with
a Western European cultural heritage, including North America).
We also suggested that the relatively high sense of “conformity” (the
restraint of actions, inclinations, and impulses likely to upset or
harm others or to violate social expectations or norms; Schwartz
et al., 2001) that characterizes Indonesians may cause them to be
relatively conservative in rating mental workload and that a rela-
tively high belief in “fate control” (a belief that life events are pre-
determined; Leung et al., 2002) in Indonesian participants also may
lead them to underestimate changes in mental effort, as self-effort is
not ascribed a large role in influencing performance. Finally, the
high power distance (the extent to which the less powerful
members of institutions expect and accept that power is distributed
unequally; Hofstede and Hofstede, 2005) and collectivism (the
extent to which individuals are integrated into strong, cohesive in-
groups groups as opposed to being expected to act individually and
care for themselves; Hofstede and Hofstede, 2005) that characterize
Indonesia may make Indonesians less willing to admit experiencing
71
increasing levels of workload because doing so would assert their
own sense of effort and set them apart from the group.
The finding that Indonesians and the Dutch use rating scales of
mental workload differently was to some extent predicted because,
as noted by Nachreiner (1999), subjective measures are likely to be
language and culture dependent, and it has been argued that
implementation of subjective measures such as the RSME in places
other than the countries where they were developed can be subject
to problems of interpretation (Van de Vijver and Poortinga, 1997).
Before concluding that cultural factors are responsible for differ-
ences in how subjective ratings of mental workload are made, it
must be ruled out that the properties of the scales used across
groups of comparison are responsible for these differences. In
general, instruments developed within one culture and used in
another culture can be subject to construct, method, or item bias
(Van de Vijver and Poortinga, 1997). Construct bias is said to occur
when there is incomplete overlap in the definition of the construct
across cultures or an incomplete coverage of the construct. Method
bias can result from different response styles, lack of comparability
of samples (e.g., differences in education, age, and gender compo-
sition), different familiarity with response procedures, or inter-
viewer effects. Finally, item bias may occur when items are
inadequately translated or formulated.
Although mental workload should be assessed with different
techniques (e.g., physiological and performance measures), in
practice, low-cost, easy to implement methods are the most used.
We therefore focus on the optimization of the RSME. The purpose of
the present study was to determine whether a more sensitive
Indonesian version of the RSME than that used by Johnson and
Widyanti (2011) could be developed by eliminating any construct
or item bias which may have been present due to the descriptive
labels and anchor points used in the original Indonesian version of
the RSME.
First, the descriptive labels of the anchor points of the English-
language RSME were newly translated into Indonesian using
back-translation (Brislin, 1970; see Fig. 1). A group of 80 Indonesian
students was then asked to match the descriptive labels to the
original nine anchor points of the RSME. Three of the descriptive
labels (“rather much”, “considerable”, and “great” effort) were
found to be inconsistently assigned to the anchor points. These
three descriptors were thus collapsed into one label capturing the
common meaning of the original three labels (the Indonesian
equivalent of “moderate” effort) and a separate group of 80 Indo-
nesian students was asked to place the descriptive labels on
a version of the RSME with only the 150-point scale marked, and no
separate anchor points. The anchor points derived from these data
were used to create a new version of the RSME (the “adapted
RSME”). The sensitivities of the original RSME and the adapted
RSME were then compared in an experiment in which participants
performed a search task in which visual displays were searched for
target items held in memory (Schneider and Shiffrin, 1977; Shiffrin
and Schneider, 1977) with different levels of difficulty (as indicated
by changes in reaction time and error rates) that has been shown to
reliably induce different levels of mental workload (e.g., Aasman
et al., 1987; Johnson and Widyanti, 2011; Rubio et al., 2004).
Groups of 50 Indonesian students each rated the subjective mental
workload associated with each condition of the search task using
either the original or the adapted RSME.
2. Materials and methods
2.1. Development of the adapted RSME
Fig. 1 gives an overview of the steps taken to create the
adapted RSME. The English-language version of the RSME
72 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76
Fig. 1. An overview of the steps taken to create the adapted RSME.
(Zijlstra, 1993) was newly translated into Indonesian following
a procedure similar to that used by Johnson and Widyanti (2011).
First, ten bilingual master’s students of the Industrial Engineering
Department of Bandung Institute of Technology (ITB; TOEFL
scores 475 or higher) performed independent forward trans-
lations of the nine descriptive labels of the RSME, presented in
order of increasing effort, into Indonesian. Following this, two
bilingual ITB faculty members (TOEFL scores 575 or higher and at
least 4 years residence in the USA) compared the ten translations
and reached consensus on the words used and their order. An
EnglisheIndonesian dictionary (Wojowasito and Poerwadarminta,
1991) was consulted where necessary. Two Ph.D. students
(TOEFL 475 or higher) then translated the scale back into English
for comparison. The two faculty members then resolved any
remaining differences in the translation. Once the RSME had been
translated into Indonesian, the adequacy of the translation was
checked by asking 80 students (mean age ¼ 19.48 years,
SD ¼ 1.19, 38 female) from the Industrial Engineering Department
of ITB to match the Indonesian descriptive labels to the original
nine anchor points of the RSME. The order of the descriptive
labels was randomized and then presented in the same,
randomized order to all participants on a sheet of paper together
with the 0e150 point (15-cm) RSME scale. Participants were
asked to draw a line from each descriptive label to one of the nine
anchor points, using each anchor point exactly once. Data from
nine participants were excluded from further analysis due to
failure to follow the instructions (e.g., matching one descriptive
label to more than one anchor point), leaving data from 71
participants for analysis.
Of the 71 participants, 43 participants (60.6%) positioned each
descriptive label of the RSME to the same anchor points as in the
English version of the scale. Eleven of the 28 participants trans-
posed the descriptive labels “rather much effort” and “considerable
effort”. Seven participants reversed the order of “considerable
effort” and “great effort”. One participant reversed the order of
“some effort” and “rather much effort”. Five participants made two
or more transpositions of three labels. The remaining four partici-
pants transposed four or five descriptive labels.
 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76 73

We next investigated whether collapsing the confusing labels

and allowing participants to choose different anchor points would
result in more consensus as to the order of the anchor points. The
descriptive labels that were subject to the most transpositions
(“rather much effort”, “considerable effort”, and “great effort”) were
collapsed into one label that was close to the meaning of the three
confusing labels (“cukupbesar” in Indonesian; “moderate effort” in
English), such that seven labels remained. A new group of 80 ITB
students (mean age ¼ 20.9 years, SD ¼ 0.8, 37 female) was asked to
place the seven descriptive labels on a version of the scale used in
the RSME that did not contain anchor points. The descriptive labels
were presented in the same, randomized order to all participants
on a sheet of paper together with the 0e150 point scale. Descriptive
labels could be placed at any position and in any order on the scale
by drawing a line from each descriptive label to a point on the scale.
One participant was excluded from analysis due to failure to follow
the instructions, leaving data from 79 participants for analysis.
The means and standard deviations of the anchor points chosen
by the 79 participants are shown in Table 1. The means were used as
the anchor points for a new Indonesian version of the RSME, which
we refer to as the adapted RSME (see Fig. 2).

2.2. Comparison of the original and adapted RSME

To determine whether the sensitivity of the adapted RSME

would be better than the original RSME, we tested Indonesian
students in a search task (Schneider and Shiffrin, 1977; Shiffrin and
Schneider, 1977) with four conditions that differed in level of
difficulty. Participants rated the subjective mental workload asso-
ciated with each condition by filling out either the original RSME or
the adapted RSME upon completion of each condition.

2.2.1. Participants
One hundred students of the Industrial Engineering Department
of ITB (mean age ¼ 20.48 years, SD ¼ 1.72, 62 female) participated
in the experiment; 50 used the original RSME to rate subjective
mental effort and 50 used the adapted RSME. Four participants Fig. 2. Adapted Rating Scale Mental effort (RSME).
were excluded from further analysis due to poor task performance,
leaving 48 in the group who used the original RSME and 48 in the
group who used the adapted RSME. there were two items in each search frame; in Condition 2 there
were two items in the memory set and search frames; in Condition
2.2.2. Task 3 there were two items in the memory set and four items in the
Participants performed a search task in which one, two, or four search frames; and in Condition 4 there were four items in both the
target items had to be held in memory while searching a visual memory set and search frames. The appropriate version of the
display of two or four items for the targets held in memory RSME was presented on the computer screen after each of the four
(Schneider and Shiffrin, 1977; Shiffrin and Schneider, 1977; see conditions and was marked by clicking a point on the scale using
Johnson and Widyanti, 2011, for details). the computer mouse.
The stimuli were presented in four conditions, varying in diffi-
culty. In Condition 1 there was one item in the memory set and
3. Results
Table 1
The anchor points of the original and the adapted RSME. 3.1. Subjective mental workload
Descriptive labels Anchor points Mean values of anchor
of the original RSME points given by Indonesian Ratings of subjective mental workload were compared using
participants (standard deviations a mixed ANOVA with group (original vs. adapted RSME) as
in parentheses) a between-subjects factor and condition (1e4, in order of difficulty)
Extreme effort 112 137 (18.6) as a within-subjects factor. As shown in Fig. 3, the effect of condi-
Very great effort 102 115 (17.6) tion was significant (F(3, 282) ¼ 36.93, p < .001, MSE ¼ 151.83), with
Great effort 85 89 (17.6)
increasing task demands being reflected in higher ratings on the
Considerable effort 71 e
Rather much effort 57 e RSME. The mean scores were numerically higher (70.76 vs. 64.06)
Moderate effort e 52 (16.0) and the range of scores numerically greater (22.19 vs. 14.67) for the
Some effort 37 e adapted version of the RSME than for the original RSME, but neither
A little effort 25 34 (13.9) the effect of group (F(1, 94) ¼ 3.265, p ¼ .074, MSE ¼ 1321), nor the
Almost no effort 13 16 (8.3)
Absolutely no effort 2 1 (2.3)
Condition  Group interaction (F(3, 282) ¼ 1.65, p ¼ .179,
MSE ¼ 151.83) were significant.
74 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76
Fig. 3. Mean RSME scores as a function of group (group using the original RSME vs.
group using the adapted RSME) and condition (in increasing order of difficulty). Error
bars represent the standard error of the mean.
3.2. Between-study comparisons
The experimental task and procedure of the current study were
identical to those used by Johnson and Widyanti (2011), making it
possible to compare the present data with those obtained in that
study. This comparison allows us to determine how stable the
results of the original RSME are for Indonesian participants and to
get an indication of how the results obtained for Indonesians using
the adapted RSME compare to those of Dutch participants using the
original RSME.
The range of values obtained with the original version of the
RSME was very similar to that of the Indonesian participants in
the Johnson and Widyanti (2011) study (mean values of 57e71 vs.
56e70 for the present and 2011 studies, respectively), and was
smaller than the range found for Dutch participants in the 2011
study (mean values of 39e70). Although cross-experiment
comparisons must be interpreted with caution, it is noteworthy
that the Condition  Group interaction (F(3, 534) ¼ 3.35, p ¼ .027,
MSE ¼ 139.35) was significant when the data from the Indonesian
participants in the 2011 study (N ¼ 84) were combined with those
of the present study and a mixed ANOVA with group (group using
original vs. group using adapted RSME) as a between-subjects
factor and condition (1e4 in order of difficulty) as a within-
subjects factor was conducted, suggesting that the adapted Indo-
nesian RSME is more sensitive to changes in workload than is the
original version of the RSME. A cross-experiment comparison of the
Indonesian participants from the current study who used the
adapted RSME and the Dutch group (N ¼ 82) from the 2011 study
by means of a mixed ANOVA with group (Indonesian group using
adapted RSME vs. Dutch group using original RSME) as a between-
subjects factor and condition (1e4 in order of difficulty) as a within-
subjects factor revealed a marginally significant Condition  Group
interaction (F(3, 384) ¼ 2.57, p ¼ .060, MSE ¼ 207.97; see Fig. 4),
suggesting that the sensitivity of the RSME is greater for Dutch than
for Indonesian participants, even when the adapted version of the
RSME is used by the Indonesians.
3.3. Search-task performance
Search-task performance was analyzed to ensure that the
performance of the two groups was similar so that differences in
estimates of mental effort could be attributed to differences in the
scales, and not differences in expended effort. The dependent
variables for the search task were accuracy (proportion correct) and
reaction time (RT). All responses that exceeded the time limit of 5 s
(0.03% of trials) were excluded from the analysis of accuracy.
Fig. 4. Mean RSME scores as a function of group (Dutch participants from study of
Johnson and Widyanti, using the original RSME vs. Indonesian participants from the
present study, using the adapted RSME) and condition (in increasing order of diffi-
culty). Error bars represent the standard error of the mean.
Incorrect responses (4.03% of trials) and RTs less than 150 ms and
greater than the condition mean plus 2.5 standard deviations,
computed separately for target-present and target-absent trials
(6.37% of correct trials), were excluded from the RT analyses.
Accuracy and RT were analyzed separately using mixed ANOVAs
with group (group using original vs. group using adapted RSME) as
a between-subjects factor and condition (1e4, in order of difficulty
due to increasing memory-set and/or search-frame size), and target
presence (target present vs. target absent) as within-subject factors.
As shown in Fig. 5, performance on the search task was similar
for both groups. The main effect of group was not significant
for either accuracy (F(1, 94) ¼ 1.272, p ¼ .262, MSE ¼ 0.006) or RT
(F(1, 94) ¼ 0.125, p ¼ .725, MSE ¼ 70,958). The effect of condition
was significant for both accuracy (F(3, 282) ¼ 88.40, p < .001,
MSE ¼ 0.002) and RT (F(3, 282) ¼ 906.32, p < .001, MSE ¼ 14,829),
with RT increasing and accuracy decreasing as a function of
increasing task difficulty. There were also significant effects of target
presence on accuracy (F(1, 94) ¼ 250.27, p < .001, MSE ¼ 0.003) and
RT (F(1, 94) ¼ 618.21, p < .001, MSE ¼ 10,494), reflecting that accu-
racy was higher and RTs were faster on target-present trials than on
target-absent trials. The Condition  Target Presence interaction was
significant for both accuracy (F(3, 282) ¼ 102.40, p < .001,
MSE ¼ 0.002) and RT (F(3, 282) ¼ 312.54, p < .001, MSE ¼ 3749),
reflecting a greater effect of condition on target-absent than on
target-present trials. No interactions involving group were signifi-
cant (all ps’ >.15).
4. Discussion
To evaluate whether the relative insensitivity of the RSME
previously found for Indonesian as compared to Dutch participants
could be due to properties of the scale, itself, a version of the RSME
with new anchor points at different locations than those of the
original RSME was developed. The original and adapted scales were
then compared in an experiment in which participants performed
a search task under four conditions of varying difficulty.
First, it was found that there was indeed some confusion about
the ordering of the descriptive labels of the original RSME scale,
especially in the middle range of the scale. The reason for this is
that the Indonesian words for “considerable”, “rather much”, and
“great effort” have similar meanings in Indonesian. To mitigate the
effects of these confusing labels, the three labels were collapsed
into one label, “cukupbesar” (in English, “moderate effort”),
and the seven labels were assigned to anchor points in a manner
similar to the original RSME. It was found that a broader range of
 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76
Fig. 5. Mean accuracy (upper panels) and mean reaction time (lower panels) as a function of target presence (target present or absent), condition (in increasing order of difficulty),
and group (group using original vs. group using adapted RSME).
the 150-point scale was used in the adapted than in the original
RSME (1e137 vs. 2e112 points for the adapted and original RSME,
respectively). Because a broader range of the scale was covered by
the anchor points, we thought it possible that the adapted RSME
might provide a more sensitive measure of mental workload for
Indonesians than the original RSME. Although the results of
a search experiment showed no significant differences in the
sensitivity of the original and adapted RSME, an examination of the
data from the present experiment combined with the data reported
by Johnson and Widyanti (2011) suggest that the adapted instru-
ment may be more sensitive than the original version.
The evidence that the adapted version of the RSME is more
sensitive than the original Indonesian version of the RSME,
although mixed, together with the suggestion (by comparison with
data from Johnson and Widyanti, 2011) that Dutch participants use
a larger range of the RSME than do Indonesians even when the
RSME has been adapted to avoid problems of translation, suggests
that the differences in the sensitivity of the mental workload
instrument as a function of cultural group are not related only to
the properties of the scale. In other words, even though the adapted
RSME appears to be an improvement on the original RSME for
Indonesian participants, Indonesians still use the scale differently
than do the Dutch. The importance of taking into account not only
language, but also culture when adapting measures of subjective
mental workload was alluded to by Hart (2006) with respect to
another popular measure of subjective mental workload, the NASA-
TLX. However, Hart reported no data regarding how culture may
influence subjective mental workload ratings.
Johnson and Widyanti’s (2011) suggestions that cultural
dimensions reflecting conformity (Schwartz et al., 2001) and fate
control (Leung et al., 2002) moderate the effectiveness of subjective
workload measuresdand that it is reasonable to assume that
workers in countries with a high power distance (Hofstede et al.,
2008) and low individualism (Hofstede et al., 2008) are likely to
be more reluctant to report increases in workloaddare supported
by the current results. The finding of a relation between individual
and cultural values and reports of mental workload supports the
75
view that culture must be taken into account in administering and
interpreting measures of mental workload (Nachreiner, 1999). For
example, in cultures in which conformity is valued and where
power distance is high, better results may be obtained when test
results are anonymous, as such individuals prefer not to stand out
from the crowd and feel uncomfortable in expressing emotions or
opinions to superiors (Hofstede and Hofstede, 2005). Thus, in
addition to taking care to equate samples as much as possible on
relevant variables, it seems advisable to measure cultural values
that influence the sensitivity of mental workload measures (e.g.,
power distance, conformity) using instruments such as the Portrait
Values Questionnaire (PVQ; Schwartz et al., 2001) and the Hofstede
Value Survey (Hofstede et al., 2008), or to use published data where
available (e.g., http://www.geert-hofstede.com/) to determine the
need to adapt procedures to measure mental load.
Our adaptation of the RSME was carried out in accord with
International Test Commission (ITC) guidelines (see Hambleton,
2001) for test development and adaptation. However, the study
was limited in that only students served as participants and only one
experimental task was used to measure mental workload. More
insight into how mental workload is reported would likely be gained
if worker populations and a broader range of tasks were used.
5. Conclusions
The adapted RSME with a reduced number of descriptive labels
appears to be used differently than the original RSME. Because
a broader range of values is used, the adapted instrument may be
more sensitive to changes in demand than the original one.
However, Dutch participants in an earlier study (Johnson and
Widyanti, 2011) appear to use an even larger range of the RSME
than do Indonesians who use either the original or the adapted
Indonesian RSME, which seems to point to fundamental differences
in how Dutch and Indonesian participants rate experienced work-
load. This fact suggests that not only scale properties, but also
culture has an influence on the measurement of subjective mental
workload in Indonesia.
76 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76

References Nachreiner, F., 1999. International standards on mental workload e the ISO 10075
series. Industrial Health 37, 125e133.
O’Donnell, R.D., Eggemeier, F.T., 1986. Workload assessment methodology. In:
Aasman, J., Mulder, G., Mulder, L.J.M., 1987. Operator effort and the measurement of
Boff, K.R., Kaufman, L., Thomas, J.P. (Eds.), Handbook of Perception and Human
heart rate variability. Human Factors 29, 161e170.
Performance. Cognitive Process and Performance, vol. II. Wiley, New York,
Ayaz, H., Shewokis, P.A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B., 2012. Optical
pp. 42-1e42-9.
brain monitoring for operator training and mental workload assessment.
Pretorius, A., Cilliers, P.J., 2007. Development of a mental workload index: a systems
NeuroImage 59, 36e47.
approach. Ergonomics 50, 1503e1515.
Brislin, R.W., 1970. Back-translation for cross-cultural research. Journal of Cross-
Reid, G.B., Nygren, T.E., 1988. The subjective workload assessment technique:
Cultural Psychology 1, 185e216.
a scaling procedure for measuring mental workload. In: Hancock, P.A., Mesh-
Cain, B., 2007. A Review of the Mental Workload Literature [online]. (Report).
kati, N. (Eds.), Human Mental Workload. Elsevier, Amsterdam, pp. 185e218.
Available from: http://www.dtic.mil/cgi-bin/GetTRDoc?AD¼ADA474193
Rubio, S., Diaz, E., Martin, J., Puente, J.M., 2004. Evaluation of subjective mental
(accessed 20.12.11.).
workload: a comparison of SWAT, NASA-TLX, and workload profile methods.
De Waard, D., 1996. The Measurement of Drivers’ Mental Workload. PhD thesis.
Applied Psychology: An International Review 53, 61e86.
University of Groningen, Groningen, The Netherlands.
Ryu, K., Myung, R., 2005. Evaluation of mental workload with a combined measure
Dey, A., Mann, D.D., 2010. Sensitivity and diagnosticity of NASA-TLX and simplified
based on physiological indices during a dual task of tracking and mental
SWAT to assess the mental workload associated with operating an agricultural
arithmetic. International Journal of Industrial Ergonomics 35, 991e1009.
sprayer. Ergonomics 53, 848e857.
Schneider, W., Shiffrin, R.M., 1977. Controlled and automatic human informa-
DiDomenico, A., Nussbaum, M.A., 2011. Effects of different physical workload
tion processing: I. Detection, search, and attention. Psychological Review
parameters on mental workload and performance. International Journal of
84, 1e70.
Industrial Ergonomics 41, 255e260.
Schwartz, S.H., Melech, G., Lehmann, A., Burgess, S., Harris, M., Owens, V., 2001.
Di Stasi, L.L., Antolí, A., Gea, M., Cañas, J.J., 2011. A neuroergonomic approach to
Extending the cross-cultural validity of the theory of basic human values with
evaluating mental workload in hypermedia interactions. International Journal
a different method of measurement. Journal of Cross-Cultural Psychology 32,
of Industrial Ergonomics 41, 298e304.
519e542.
European Organization for the Safety of Air Navigation, 2003. Review of Workload
Shiffrin, R.M., Schneider, W., 1977. Controlled and automatic human information
Measurement, Analysis and Interpretation Methods [online]. (Report). Available
processing: II. Perceptual learning, automatic attending and a general theory.
from: http://www.eurocontrol.int/integra/gallery/content/public/documents/
Psychological Review 84, 127e190.
expt_def_wp2_final.pdf (accessed 20.12.11.).
Van de Vijver, F.J.R., Poortinga, Y.H., 1997. Towards an integrated analysis of bias in
Gopher, D., Donchin, E., 1986. Workload e an examination of the concept. In:
cross-cultural assessment. European Journal of Psychological Assessment 13,
Boff, K.R., Kaufman, L., Thomas, J.P. (Eds.), Handbook of Perception and Human
29e37.
Performance. Cognitive Process and Performance, vol. II. Wiley Interscience,
Van Doorn, L., Zijlstra, F.R.H., 1988. Variation in response functions complicates the
New York, pp. 41-1e41-49.
evaluation of scales. In: Saris, W.E. (Ed.), Variation in Response Functions:
Hambleton, R.K., 2001. The next generation of the ITC test translation and adap-
a Source of Measurement Error in Attitude Research. Sociometric Research
tation guidelines. European Journal of Psychological Assessment 17, 164e172.
Foundation, Amsterdam, pp. 87e97.
Hart, S.G., 2006. NASA-Task Load Index (NASA-TLX); 20 years later. In: Proceedings
Veltman, J.A., Gaillard, A.W.K., 1996. Pilot workload evaluated with subjective and
of the Human Factors and Ergonomics Society 50th Annual Meeting. Human
physiological measures. In: Brookhuis, K.A., Weikert, C., Moraal, J., De Waard, D.
Factors and Ergonomics Society, Santa Monica, CA, pp. 904e908.
(Eds.), Aging and Human Factors. Proceedings of the Europe Chapter of
Hart, S.G., Staveland, L.E., 1988. Development of NASA-TLX (Task Load Index):
the Human Factors and Ergonomics Society Annual Meeting 1993. University
results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N.
of Groningen, Traffic Research Centre, Haren, pp. 107e128. Retrieved from:
(Eds.), Human Mental Workload. Elsevier Science Publisher, North-Holland,
http://www.hfes-europe.org/books/agibk.htm.
pp. 139e184.
Wilson, M., Smith, N.C., Chattington, M., Ford, M., Marple-Horvat, D.E., 2006. The
Hofstede, G., Hofstede, G.J., 2005. Cultures and Organizations: Software of the Mind,
role of effort in moderating the anxiety e performance relationship: testing the
second ed. McGraw Hill, New York.
prediction of processing efficiency theory in simulated rally driving. Journal of
Hofstede, G., Hofstede, G.J., Minkov, M., Vinken, H., 2008. Value Survey Module
Sports Sciences 24, 1223e1233.
2008 Manual [online]. Available from: http://www.geerthofstede.nl/research–
Wierwille, W.W., Eggemeier, F.T., 1993. Recommendations for mental workload
vsm/vsm-08.aspx (accessed 20.12.11.).
measurement in a test and evaluation environment. Human Factors 19,
Johnson, A., Widyanti, A., 2011. Cultural influences on the measurement of
263e281.
subjective mental workload. Ergonomics 54, 509e518.
Wojowasito, S., Poerwadarminta, W.J.S., 1991. KamuslengkapInggeris-Indonesia
Lin, Y., Cai, H., 2009. A method for building a real-time cluster-based continuous
Indonesia-Inggerisdenganejaan yang disempurnakan (English-Indonesian,
mental workload scale. Theoretical Issues in Ergonomics Science 10, 531e543.
Indonesian-English dictionary). Hasta, Bandung.
Leung, K.L., Bond, H.M., De Carrasquel, S.R., Munoz, C., Hernandez, M., Murakami, F.,
Zijlstra, F.R.H., 1993. Efficiency in Work Behavior: a Design Approach for Modern
Singelis, T.M., 2002. Social axioms: the search for universal dimensions of
Tools. Doctoral thesis. Delft University of Technology, Delft, The Netherlands.
general beliefs about how the world functions. Journal of Cross-Cultural
Retrieved from: http://repository.tudelft.nl/view/ir/uuid%3Ad97a028b-c3dc-
Psychology 33, 286e302.
4930-b2ab-a7877993a17f/.
Mulder, L.J.M., Dijksterhuis, C., Stuiver, A., De Waard, D., 2009. Cardiovascular state
Zijlstra, F.R.H., Van Doorn, L., 1985. The Construction of a Scale to Measure Perceived
changes during performance of a simulated ambulance dispatchers’ task:
Effort. Technical Report. Delft University of Technology.
potential use for adaptive support. Applied Ergonomics 40, 965e977.