IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT, VOL. 7, NO.
4, DECEMBER 2015
Beyond Subjective Self-Rating: EEG Signal
Classification of Cognitive Workload
Pega Zarjam, Julien Epps, Member, IEEE, and Nigel H. Lovell, Fellow, IEEE
Abstract—Cognitive workload is an important indicator of
mental activity that has implications for human–computer in-
teraction, biomedical and task analysis applications. Previously,
subjective rating (self-assessment) has often been a preferred mea-
sure, due to its ease of use and relative sensitivity to the cognitive
load variations. However, it can only be feasibly measured in a
post-hoc manner with the user’s cooperation, and is not available
as an online, continuous measurement during the progress of the
cognitive task. In this paper, we used a cognitive task inducing
seven different levels of workload to investigate workload dis-
crimination using electroencephalography (EEG) signals. The
entropy, energy, and standard deviation of the wavelet coefficients
extracted from the segmented EEGs were found to change very
consistently in accordance with the induced load, yielding strong
significance in statistical tests of ranking accuracy. High accuracy
for subject-independent multichannel classification among seven
load levels was achieved, across the twelve subjects studied. We
compare these results with alternative measures such as perfor-
mance, subjective ratings, and reaction time (response time) of
the subjects and compare their reliability with the EEG-based
method introduced. We also investigate test/re-test reliability of
the recorded EEG signals to evaluate their stability over time.
These findings bring the use of passive brain-computer interfaces
(BCI) for continuous memory load measurement closer to reality,
and suggest EEG as the preferred measure of working memory
load.
Index Terms—Cognitive workload, electroencephalog-
raphy (EEG), multichannel classification subject-independent,
self-rating, wavelet coefficients.
I. INTRODUCTION
I N BRAIN-COMPUTER INTERFACES (BCI), human–
computer interaction (HCI), cognitive neuroscience,
biomedical engineering, and psychology researchers have an
ongoing interest in cognitive monitoring [1]–[3], in order to
Manuscript received November 12, 2014; revised February 26, 2015; ac-
cepted May 22, 2015. Date of publication June 04, 2015; date of current version
December 09, 2015.
P. Zarjam is with the School of Electrical Engineering and Telecommunica-
tions, University of New South Wales, Sydney, NSW 2052, Australia (e-mail:
p.zarjam@unsw.edu.au).
J. Epps is with the School of Electrical Engineering and Telecommunications,
University of New South Wales, Sydney, NSW 2052, Australia, and also with
the ATP Research Laboratory, National ICT Australia, Eveleigh, NSW 2015,
Australia (e-mail: j.epps@unsw.edu.au).
N. H. Lovell is with the Graduate School of Biomedical Engineering,
University of New South Wales, Sydney, NSW 2052, Australia (e-mail:
n.lovell@unsw.edu.au).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TAMD.2015.2441960
1943-0604 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
301
gain information on the cognitive user’s state such as their
mental or cognitive activities, memory load and task engage-
ment. These measures are criteria for how hard a user is
working to solve a problem or use an interface, and can be
employed to support the user’s goals or adapt an interface [3].
Cognitive monitoring, also called passive BCI, can detect the
potential for human error or the loss of control over a system
by providing information on the user’s state [1]. Cognitive load
is the load imposed on the working memory during a cogni-
tive process [4], and its measurement is of great importance in
cognitive monitoring [3], [4]. This is to avoid cognitive/mental
overload of the user or interface and maintain efficiency and
productivity in work performance, especially in critical or high
mental load workplaces such as air traffic control, military op-
erations, medical and emergency applications. A mechanism of
interest in these fields is working memory, which is used for
short-term retention, utilization, and manipulation of informa-
tion in the mind [5]. More broadly, changing characteristics of
working memory with memory load or task engagement [4], and
the relationship with the cortical activation are areas of growing
study in recent years [6], [7].
There are different techniques available to measure cognitive
load, including subjective measures (e.g., performance-based,
subjective ratings, reaction time), and behavioral and phys-
iological methods. Among subjective measures, self-rating
(self-report) has been widely used as the preferred measurement
method [8], [9], which consists of posttask questionnaires. This
is because these are easy and inexpensive to administer and
assess, they do not obtrude with primary task performance, and
they can detect small variations in workload with a relatively
good sensitivity [4]. However, subjective measures cannot
provide satisfactory results in all scenarios as they basically
rely on the assumption that the subjects are willing and able
to respond, accurately [10]. Another limitation is that they can
only be measured in a post-hoc manner and are not available
as an on-line, continuous measurement during the progress
of the cognitive task, which is the case for many real-time
BCI/HCI applications. Behavioral methods (e.g., speech,
mouse/pen-input movements [11]–[13]) can also reflect cogni-
tive load, but they are the most distant level of measurement
from the cognitive activity, and their sensitivity to workload
is mostly not high. Physiological methods include heart rate
variability [14], eye movement [15], hormone levels [10], skin
conductance/galvanic skin response (GSR) [16], [17], and brain
activity. Among these measures, monitoring the brain activity
has been recognized as the most sensitive and consistent re-
flector of cognitive load [2]. However, physiological methods
302 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT, VOL. 7, NO. 4, DECEMBER 2015
generally have their own constraints, for example requiring
equipment, set up, and exhibiting susceptibility to artifact.
Electroencephalography (EEG) can be applied in online and
continuous measurement, detecting subtle variations, particu-
larly in instantaneous load in cognitive processing [2]. It is also
reliable, noninvasive, inexpensive (compared with the peer neu-
roimaging devices), and offers the best temporal resolution [2].
The usage of EEG has also become more feasible for real-world
applications with the availability of wireless EEG systems, re-
cently [18]. Hence, what is needed is investigation of the sen-
sitivity and reliability of EEG-derived indicators that can esti-
mate workload in or near real-time, and comparison with ex-
isting methods.
While previous researchers have investigated classifying
working memory load with EEG, they mostly studied cog-
nitive processes and memory performance when executing
memory-associated cognitive tasks such as semantic, episodic,
or recognition memory tasks [7], [19]–[24]. By contrast, the
classification of other cognitive tasks associated with general
task demands such as task difficulty and attentional demands
have been relatively less studied, in which the largest number
of discrete task load levels imposed has been five levels [25],
[26] so far. In most of these studies, power-based features such
as signal’s maximum or average powers extracted from its
power spectrum density (PSD) have been used [3], [24], [25],
[27], [28]. This is due to their ease of use and interpretation
of results and computational speed [29]. Other features such
as auto-regressive models [30] and subband entropy [27] have
also been applied, however the applicability of many further
features is still yet to be examined.
To investigate these questions further, we previously de-
signed an arithmetic task with seven levels of difficulty, to
advance the granularity of load levels induced and examine the
suitability of other features for discriminating between varying
fine levels of induced working memory load [31]. Thereby,
wavelet-entropic and phase synchronisation features were
studied, and the median of the extracted features declined as
workload increased, indicating that the EEG signal’s disorder
decreased and the synchrony of the responsive neural groups
increased, with increasing workload.
In the current study, we examine a new set of features, based
on the discrete wavelet transform (DWT), that could be used
for near real-time applications. The reliability and sensitivity of
this EEG-based method is also compared with common subjec-
tive measures of cognitive load; i.e., performance (accuracy of
responses), subjective ratings, and reaction time of the subjects.
The performance of the best performing feature in [31] is also
compared with the new features, herein.
The significance of this research is summarized as follows:
first, the knowledge of suitable features extracted from the EEG
signals for discriminating between fine induced load levels is
extended. Second, we represent results that EEG can outper-
form self-ratings, which is an interesting result for empirical
cognitive load research, in general. Third, we assess test/re-test
reliability of the recorded EEG signals to evaluate their sta-
bility over time, which is required for the development of an
EEG-based memory load measurement method to be used in
real-time BCI/HCI settings [32]. This work is an extension of
TABLE I
DETAILS OF THE TASK DESIGN FOR ALL SEVEN LEVELS. IN EACH TASK
LEVEL, SIX ADDITIONS WERE PRESENTED [31]
our previous work in terms of feature extraction and advances
the existing knowledge of accurate methods for cognitive work-
load discrimination, in general.
II. EXPERIMENTAL METHODS
A. Task
The choice of an arithmetic task was based on the need to
induce seven different load levels in a consistent manner. When
performing calculations on multidigit numbers, several steps
are involved: retrieving an arithmetical fact from long term
memory (LTM), storing interim results/temporary information
(in short-term memory (STM), closely related to working
memory) and doing calculation strategies (extra operations,
e.g., carrying or borrowing) which apply considerable demand
on working memory [33], [34].
The details of the task have been presented in [31]; briefly, it
is an addition task with seven levels of difficulty, starting from
one digit addition (very easy) to multidigit addition (extremely
difficult). The task was displayed and controlled on a laptop PC
with a viewing distance of 70 cm to the subject. Each number
was shown at the centre of the screen in Arabic notation for
3 seconds. Subjects were asked to add the two presented num-
bers (shown sequentially), then were given 2 s (blank page) for
retention followed by a multiple choice menu that presented the
possible answers. There were a total of 42 addition problems,
in seven difficulty levels, each level lasting for 2 min. Each task
duration was chosen to be short and was followed by a short
break (30 s) to ensure that subjects did not become fatigued
[35]. The difficulty level was manipulated by varying the n-digit
numbers used and carries required to calculate the addition. The
possible answers given in the multiple choices were selected
carefully to be very close to the correct answer (e.g., with for-
gotten carry if needed or inappropriate carry) to prevent the sub-
jects from guessing the answer and ensure that they performed
the calculation (i.e., were engaged in the task). The given time
for each addition problem in all the levels was the same, to im-
pose a greater mental task load on the subject when performing
the more difficult addition. There was also a time limit for an-
swering each addition task and feedback on the difficulty level,
in which the subjects rated the task difficulty experienced for the
given addition task after each task. The experiment was run in
order starting from very easy (very low level) to extremely dif-
ficult (extremely high level). The task detail is shown in Table I.
Besides recording the EEG signals and subjective ratings, we
gathered the reaction times (from when the possible answers
were displayed until one answer was chosen) and performance
ZARJAM et al.: BEYOND SUBJECTIVE SELF-RATING: EEG SIGNAL CLASSIFICATION OF COGNITIVE WORKLOAD
accuracy for each addition task. To minimize any muscle move-
ment artifact due to electromyogram (EMG) activity during the
recording, subjects were asked to avoid any unnecessary phys-
ical movements and their hand was placed in a fixed position,
where they could still make slight finger movements in response
to the correct answer on the mouse. Since the channels in the
frontal lobe are susceptible to ocular artifact (Electrooculog-
raphy/EOG), subjects were required to refrain from excessive
blinking as much as possible. The subjects were given 30 sec-
onds break between each two levels, when they could move or
blink. In the baseline (rest) condition, conducted after the exper-
iment, the subjects were asked to sit relaxed and keep their eyes
closed for 2 min. We chose closed-eyes as a baseline to control
the rest condition more and ensure the subjects were in a relaxed
position.
B. Subjects
Twelve healthy male volunteers engaged in postgraduate
study participated in the experiment. Their age ranged from
24–30 years, all were right-handed and had normal or corrected
to normal eye-sight. They gave written informed consent, in
accordance with human research ethics guidelines. The subjects
were asked to refrain from taking alcohol and caffeine 12 h
prior to the experiment. The experiment (test) was done in
one session and the task lasted about 15 min. The experiment
was repeated (retest session) after a six-month interval for two
subjects.
C. EEG Recording
The subjects’ EEG signals were recorded using an Active
Two acquisition system, at the ATP Laboratory of National ICT
Australia in Sydney. The experiment was conducted under con-
trolled conditions in an electrically isolated lab, with a min-
imum distance of 5 m from power sources to the experiment
desk and under natural illumination. Each recording contained
32 EEG channels mounted in an elastic cap, according to the ex-
tended international 10–20 system. A linked earlobe reference
was used and impedance was kept under k . The EEG signals
were passed through a band-pass filter with cut-off frequencies
of Hz and were recorded at a 256 Hz sampling rate.
III. ANALYSIS METHODS
A. Preprocessing
In order to obtain artifact-free data, the acquired EEG signals
were first visually inspected and segments contaminated with
EMG and EOG artifacts were removed. For the purpose of fea-
ture extraction, the EEG signals were segmented using a rect-
angular window of length seconds. After removing the
DC level of the segments, we resampled them at Hz
to reduce the computational load of the feature extraction algo-
rithms. Finally, the EEG segments were band-pass filtered
in the frequency band of Hz (as the EEG signals gen-
erally do not have many useful frequency components above
30 Hz [36]). Hereafter, we denote the EEG segment under anal-
ysis by which contains samples per seg-
ment, with no overlap between the successive segments.
303
TABLE II
EEG FREQUENCY BANDS APPROXIMATELY CORRESPONDING TO EACH
WAVELET SCALE, Hz [31]
B. Spectral Analysis
In order to quantitatively study the PSD of the EEG signals
for this task in a summarized, comparative manner across all
channels, we calculated the spectral edge frequency (SEF) of
the signals recorded from all 32 channels. Here, the s were con-
sidered as the frequency below which 95% of the EEG signal
power resides.
C. Multi-Resolution Analysis Using the DWT
We employed a new set of features, using the discrete wavelet
transform (DWT). The DWT has proven to be one of the most
suitable tools for processing of the EEG signals, since it does
not assume the signals are stationary and offers an optimal
time-frequency resolution for analysis of these signals which
are generally dominated by low frequencies [37] and localized
signal representation in the time- and frequency-domain for
EEG signals analysis [38]. The efficient DWT computational
structure is obtained by wavelets and scaling sequences de-
duced from one octave to the next by a two-scale difference
equation. At each octave level ,
a signal is passed through a low-pass filter and a
high-pass filter to produce approximate (i.e., ) and detail
(i.e., ) coefficients, respectively [39]. The choice of the
mother wavelet depends on the application. The smoothing
feature of the Daubechies wavelet makes it suitable to detect
variations in the EEG signals [39]. Therefore, we selected the
Daubechies-4 mother wavelet and the parameter was chosen
to be 3 resulting in 3 wavelet scales, corresponding approxi-
mately to the EEG frequency bands, as shown in Table II.
D. Feature Extraction
During preliminary analyses, we extracted many features
from our previous work for classifying three levels of load,
for example zero-crossing rate, maximum cross-correlation,
spectral coherence, instantaneous frequency, phase locking
value, intensity weighted mean frequency, intensity weighted
bandwidth and [40]–[42]. These features were unable to
discriminate all seven load levels accurately.
We then tested a series of wavelet-based features, of which
the entropy, energy, and standard deviation of the coefficients
were best able to distinguish the seven load levels. We extracted
304 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT, VOL. 7, NO. 4, DECEMBER 2015
these features from each of 13 artifact-free EEG segments (from
each channel), since the total artifact-free task duration was at
least 65 s across all load levels and subjects. For each wavelet
scale , wavelet approximate and detail coefficients are given
by , respectively, where
is the number of coefficients in scale . The fea-
tures employed were calculated using the wavelet coefficients
as follows.
1. Entropy of the approximate coefficients ( ): The
first feature is the Shannon entropy of the approximate co-
efficients and is given by [31]
(1)
From the above equation, it is clear that
. The feature is zero when only one of the wavelet
coefficients is nonzero and attains its peak value when all
coefficients are equal. This feature therefore captures the
variability and self-similarity among the coefficient values,
and is also a measure of the flatness of the approximate
coefficients. Analysis of the effects of increasing the load
level on the approximate coefficients showed that as the
load level increases, the number of significant coefficients
decreases, which results in lower values for the entropy
given in (1).
2. Energy of the approximate coefficients (WSTD): This
feature represents the energy of the EEG segment under
analysis in a particular frequency band. The can
be calculated as the sum of the squares of the wavelet ap-
proximate coefficients for that particular scale
(2)
The value of this feature shows how the signal energy
is distributed over different subbands. From the spectral
analysis in Section IV-A, it is known that the signal com-
ponents with the largest power are in the low frequency
bands, therefore we expect this feature provides some
useful information in higher wavelet scales (e.g., scale
).
3. Standard deviation of the approximate coefficients
( ): This feature shows how much the EEG seg-
ment under analysis varies from the mean.
E. Statistical Analysis of Extracted Features
We used a Kruskal–Wallis test to measure how effective the
features were in separating different workloads imposed, since
there are more than three independent load level groups; it is
a nonparametric method for one-way analysis of variance and
is not affected by variations in small portions of the data. Per-
forming the test examines the hypothesis that all samples were
taken from identical populations and is especially sensitive to
differences in central tendency [43]. This test was suggested in
previous related studies [27], [44], due to the ranked nature of
the data. We calculated -values by Kruskal–Wallis test across
all the channels for each extracted feature, for each participant.
To assess the consistency of ranking among the task loads, we
also conducted Friedman’s test, which ranks each row together,
then considers the values of ranks by columns [43]. Generally,
the Friedman test is a nonparametric statistical test which exam-
ines differences between groups when the dependent variable
under measurement is ordinal. This test is used as column effects
are of interest in the study and it examines only column effects
after adjusting possible row effects [43]. This is an important
test for measurement applications like cognitive load, where an
ordinal relationship between load levels must be consistently
observed in order for EEG features to be meaningful indicators.
Here, we considered the designed task loads as columns and the
subjects as rows. This ranked the measured load levels for all
the subjects, each time for one EEG channel (for one extracted
feature).
F. Feature Selection
In line with the earlier observation of the importance of an
ordinal relationship between load levels, all feature/channel/
frequency band combinations that exhibited a consistent
monotonic trend in the same direction across all subjects were
selected.
G. Classification
We classified the imposed memory loads using the multilayer
perceptron (MLP) structure of artificial neural networks (ANN)
from the EEG signals recorded when performing the task. ANNs
are known as flexible classifiers and with careful selection of ar-
chitecture can be a great choice for classifying noisy and nonsta-
tionary data such as EEGs [45]. We used an MLP structure for
the ANN, which has the ability to handle complex and nonlinear
input-output relationship with hidden layers. A four-layer archi-
tecture ANN with two hidden layers was adopted here based
on empirical results; the first hidden layer included 20 neurons,
the second hidden layer 14 neurons and the output layer 7 load
levels.
IV. EXPERIMENTAL RESULTS
A. Spectral Analysis
The results of the analysis are shown in Fig. 1, where for
each level of task difficulty, the median of the SEFs of the sig-
nals recorded from a particular channel is plotted. This shows
that most of the power of the signals recorded from the frontal
channels (the frontal left; channels Fp1, AF3, F7, F3, FC1, FC5
and the frontal right; channels FC6, FC2, F0.01 , F8, AF4,
Fp2) lies in the lower EEG frequency band, corresponding to
the delta frequency band. For the occipital lobe channels (PO3,
O1, Oz, O2, PO4) there is a significant power spread up to 12 Hz
which includes the delta, theta and alpha frequency bands. The
left hemisphere lobe (temporal, central and parietal channels,
i.e., T7, C3, CP1, CP5, P7, P3, Pz), displays significant fre-
quency components up to 8 Hz which include the delta and theta
frequency bands. The right hemisphere lobe (temporal, cen-
tral, parietal channels, i.e., P4, P8, CP6, CP2, C4, T8) includes
significant frequency components in the delta, theta and alpha
ZARJAM et al.: BEYOND SUBJECTIVE SELF-RATING: EEG SIGNAL CLASSIFICATION OF COGNITIVE WORKLOAD
Fig. 1. Median of the extracted spectral edge frequency from the segmented
EEG for 32 EEG channels across twelve subjects for different load levels im-
posed. The lowest task load is denoted by L1 and the highest by L7. This shows
the frequency range within which the concentration of the signals’ power lies,
across different regions of the brain (frontal, occipital, left/right hemisphere) for
different load levels. Based on this, the various EEG channels were studied in
their corresponding frequency ranges (see Table IV).
TABLE III
CHANNELS THAT SHOWED THE SMALLEST -VALUES CALCULATED BY
KRUSKAL-WALLIS TEST, ACROSS ALL SUBJECTS
frequency bands. Clearly, the right hemisphere lobe exhibits
higher frequency components compared with its left counter-
part in Fig. 1.
B. Statistical Analysis of Extracted Features
Statistical analysis of the extracted features across all the sub-
jects showed that all of the channels return small -values (lower
than level). Table III displays the channels that returned the
smallest calculated -value results for each feature, across all
subjects. As the -values suggest, the , , and
features reveal a strong statistical significance in dif-
ferentiating the seven task load levels imposed, across all the
EEG channels, for twelve subjects.
The channels whose ranks were common across all subjects
(i.e., consistent monotonic trend) for the extracted feature were
as follows: F7, F3, FC5, C3, CP5, O1, CP6, C4, T8, FC6, FC2,
and AF4. For instance, Friedman’s test result for channel AF4
for the feature was: mean-ratings of 2.66, 2.75, 3.41,
3.75, 4.08, 5.00 and 6.33 for ranking task difficulty levels from
1 to 7, respectively, with . As Table IV shows, the frontal lobe is the most active region,
C. Feature Selection
The features defined in Section III-D were extracted from
each raw EEG segment of seconds’ length for all EEG
channels of each subject. We investigated these features for
different channels in their associated frequency bands according
to the findings of Section IV-A. The results are summarized in
Table IV, for those channels which displayed a consistent (mono-
305
Fig. 2. Medians of extracted from segmented EEG data in the delta
band, from the frontal lobe of subject 1. In many frontal EEG channels, the
median decreases as workload increases.
TABLE IV
EEG CHANNELS, BANDS AND FEATURES FOR WHICH A CONSISTENT
DECREASING TREND WAS OBSERVED ACROSS ALL TWELVE SUBJECTS. BASED
ON THESE RESULTS, THE MOST SUITABLE CHANNELS ARE SELECTED
tonic) trend during task load variation across all subjects. For
illustration purposes, Fig. 2 displays the median of the extracted
feature from frontal lobe channels in scale 3, for subject
1. The median of the feature is able to distinguish the
seven task loads well, across most of the EEG frontal channels
in the delta frequency band, consistently showing decreasing
median with increasing task difficulty. On the other hand, the
medians of the feature of the frontal lobe decrease
consistently in some channels in the delta band when the task
difficulty increases, as shown in Fig. 3. By the same token, the
medians of the feature show a similar decreasing trend.
reflecting the largest number of channels contributing to the
task load discrimination for each feature extracted.
D. Classification
As mentioned in Section IV-B, for the extracted features all
32 channels provided a low -value, showing statistical signif-
icance in distinguishing the load levels. Fewer channels exhib-
ited consistent trends across all twelve subjects, as displayed in
306 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT, VOL. 7, NO. 4, DECEMBER 2015
Fig. 3. Boxplot of the extracted from the segmented EEG data for
channel F3 for subject 1 in the delta band. On each box, the red mark is the
median; the edges of the box are the 25th and the 75th percentiles. Very low
level is denoted by L1 and the extremely difficult level by L7.
TABLE V
ACCURACY OF 7-CLASS CLASSIFICATION FOR THE EXTRACTED FEATURES
BY ANN USING LEAVE-ONE-OUT TECHNIQUE, ACROSS TWELVE SUBJECTS,
COMPARED TO ACCURACY OF 7-CLASS CLASSIFICATION FOR SUBJECTIVE
SELF-RATING
Table IV. To gauge the classification performance of the pro-
posed features and also narrow down the number of channels
under study, we considered those features which not only re-
vealed low -values but also a consistent trend (in Table IV)
from frontal EEG channels, across all subjects.
We applied leave-one-out cross-validation for training and
testing data (using one subject as the validation data, and the
remaining eleven subjects as the training data for each fold),
in a subject-independent arrangement. These selected channels
for each feature were then input to an ANN classifier in a
multichannel manner. This multichannel approach offers the
benefit of precisely defining the set of brain areas responsible
for producing the EEG signal which cannot be done by a single
channel, because of electrical diffusion in the physiological
tissues [46]. Although it would be desirable, we could not
use a separate data partition for feature and channel selection
due to the relatively small size of the database. The ANN per-
formed a 7-class classification on the selected channels listed
in Table IV. The results are shown in Table V, from which the
selected frontal channels reveal a high classification rate for the
extracted , and features.
E. Comparison with Performance, Reaction Time and
Subjective Ratings
We collected the reaction times and the accuracy of responses
(performance) to each addition task. We observed that the in-
crease in task difficulty led to longer reaction times according to
task difficulty level, mostly from load level 2 onwards for each
subject. However, eleven out of twelve subjects experienced a
Fig. 4. Averaged reaction time versus task difficulty across twelve subjects.
On each box, the red mark is the mean; the edges of the box are the 25th and
the 75th percentiles [31].
TABLE VI
RESPONSE ACCURACY (TASK PERFORMANCE) PERCENTAGE VERSUS TASK
DIFFICULTY AVERAGED OVER TWELVE SUBJECTS AND FRIEDMAN’S TEST
RESULTS
TABLE VII
CONFUSION MATRIX FOR SELF-RATINGS RESULTS ACROSS TWELVE SUBJECTS.
EACH LOAD LEVEL (TASK LOAD) REPRESENTS 72 ADDITION TASKS
longer reaction time at the beginning of the experiment, i.e., for
level 1 compared with level 2. This might be due to the time
these subjects needed to adjust to the experimental procedure.
It was also observed that the reaction time consistently increases
for most subjects as the task difficulty grows.
We also averaged the reaction times over twelve subjects for
each task difficulty level. As shown in Fig. 4, the average reac-
tion time increases consistently as the task difficulty increases.
The reaction time considerably increases in load level 7, the
highest load level, in which three-digit numbers with three car-
ries are introduced.
Analysis of the collected accuracy of responses across twelve
subjects revealed that from load level 2 onwards, the percent
accuracy decreased consistently as task difficulty increased (see
Table VI). In order to validate the accuracy of responses statisti-
cally, we applied Friedman’s test. The designed task loads were
considered as columns and the responses as rows for twelve sub-
jects. This ranked the given responses according to load levels
for all the subjects. This result is also given in Table VI.
Table VII shows the self-rating confusion matrices across
twelve subjects. Each row indicates the task load imposed and
each column the task load reported by the subjects. The shaded
ZARJAM et al.: BEYOND SUBJECTIVE SELF-RATING: EEG SIGNAL CLASSIFICATION OF COGNITIVE WORKLOAD
Fig. 7. Medians of the entropy of the approximate coefficients extracted from
the segmented EEG data in the delta band in retest session, from the frontal lobe
of the same subject shown in Fig. 3. In most of the frontal EEG channels, the
decreasing trend with task load increase is preserved.
areas show the load levels correctly rated by the subjects. As
seen, the highest numbers of correct ratings belong to L1 to
L3, indicating that subjects were mostly able to rate the lower
load levels, correctly. On the other hand, adjacent intermediate
levels were often confused, for instance level 2 was confused
with level 1 with the highest rate and level 4 with level 3. Ap-
parently, subjects tended to under-rate the task difficulty experi-
enced in most cases. The results demonstrate that the self-ratings
do not present a consistent trend corresponding to the task load
imposed. Obviously, our subjects seemed not very capable of
rating their experienced mental burden, even though they were
advised prior to the task that there are 7 load levels to rate, in-
crementally. We also applied Friedman’s test to the collected
subjective ratings. Here, we considered the designed task loads
as columns and the subjective ratings as rows for twelve sub-
jects. This ranked the reported load levels for all the subjects, as
follows: the mean-ratings of; 4.62, 4.91, 5.04, 3.12, 3.70, 2.87,
and 3.70, for ranking task difficulty levels from 1 to 7, respec-
tively, with . Clearly, the ranking is not consistent
and is not significant at a 0.01 level. Accuracy of 7-class classi-
fication for this self-rating is shown in Table V.
F. Re-Test Analysis
We analyzed the recorded data in two ways in order to assess
the test/re-test reliability of the feature distributions. First, a one
way repeated ANOVA analysis was used to examine any differ-
ences across the two sessions of test and retest [32]. This test can
be used for both parametric and nonparametric data and when
we have a single group on which we have measured something
a few times (test/re-test in our case) [43]. Second, we checked
the trends shown by the features extracted from each channel
when increasing the task load, as discussed in Section IV-C.
Using ANOVA, there were no significant effects between
sessions for the recorded EEGs across two corresponding load
levels in test and retest (for the same channels and subjects). For
instance, for the feature extracted from Ch4 (F3) between L1
from the test and L1 from the retest session, we found
and also between L7 from the test and L7 from the retest session
. As shown by Table VI, the study of subjects’ performance
We also repeated the procedure conducted in Section IV-C for
the retest EEG signals. For illustration purposes, Fig. 7 shows
307
the extracted feature from the frontal channels of sub-
ject 1. As shown, the increasing trend of the signal as the task
load increases was preserved in the retest session as well, con-
firming the Fig. 3 results. This indicated the cognitive load EEG
signals are reproducible for high load cognitive tasks even after
a 6-month interval.
V. DISCUSSION
We evaluated the usefulness of the proposed multiresolution
feature set and sensitivity of the EEG signals in discriminating
cognitive load in fine seven levels; first, using two statistical ap-
proaches (Kruskal-Wallis and Friedman tests) in Section IV-B,
second, using classification accuracy across twelve subjects in
Section IV-D. The results showed that high task load discrimi-
nation was mainly achieved in the frontal lobe of the brain. This
is supported by related findings emphasizing that the frontal
lobe is necessary for maintaining and carrying out calculation
tasks, due to its tight association with attention and working
memory [31], [48], [49].
The decrease of the extracted entropy and energy of the ap-
proximate coefficients with the task difficulty increase could be
interpreted as narrower distribution of the entropy or energy of
the coefficients, in the frequency band under study. This may
indicate that the brain behaves in a more certain and focused
manner when performing more difficult tasks [47]. Our study
not only affirms the literature but also adds that in the case of in-
creasing task difficulty; the frontal lobe is affected more deeply.
This might be interpreted as the subjects paying more attention
or becoming more focused while tackling more difficult tasks.
The feature set used herein is based on wavelet coefficients,
which have shown their capability for EEG signal classification
in pathological diagnoses previously [50], [51], but had not been
used in the area of cognitive load or mental task classification,
prior to this study. Only in a few related studies have wavelet co-
efficients been deployed: to remove artifacts from EEG data in
[26] and discriminate three load levels on the basis of the appear-
ance time and total power extracted from the EEG signals [27].
This feature set extends our current knowledge of successful
wavelet-features for an EEG-based load classification system
[31] and advances the existing knowledge of EEG-based fea-
tures, in general.
In this research, we put an emphasis on the delta band due to
its role in the brain region of interest, i.e., the frontal lobe and its
close relationship with working memory [48], [49], [52]. This
frequency band has been investigated in only a couple of pre-
vious studies, which showed that by increasing task demand,
subject attention to the task and also the delta band activity in-
creases [53], [54]. This frequency band has attracted less atten-
tion, perhaps due to concerns of artifact contamination [54]. We
point out that, our task was conducted under controlled condi-
tions and we eliminated DC level and artifact-contaminated seg-
ments and studied only artifact-free segments [31]. However, in
informal experiments we have previously extracted features like
those in this paper from artifact-affected data, and not observed
any degradation in classification accuracy.
indicated that as the load level increased, the response accu-
racy consistently declined. This was confirmed statistically by
308 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT, VOL. 7, NO. 4, DECEMBER 2015
a low -value in Section IV-E. Table VII also showed that the
subjects were not capable of accurately rating their experienced
mental burden (self-ratings). This is despite the fact that prior
to the experiment, subjects were instructed that the task will
be presented in an increasing seven-level manner. Their given
self-ratings were not statistically significant by Friedman test,
which confirms doubts expressed previously about the relia-
bility of self-ratings as an indicator of the mental burden im-
posed [55], [56]. Self-rating is well-established as one of the
subjective assessment methods for cognitive load [4], [57]. This
reduced sensitivity of self-ratings to workload variations might
be interpreted in terms of cases when the task demand lies in ei-
ther the low load (underload) or the high load (overload) levels,
in which there is a trade-off between the amount of invested re-
sources and time-sharing requirements on the working memory
imposed by the task demand [55], [56]. This trade-off may cause
disassociation between the subjects’ performance and self-rat-
ings [55], [56].
Our results therefore demonstrate that cognitive load assess-
ment by EEG can outperform self-ratings, which is an inter-
esting result for empirical cognitive load research in general.
Therefore, it can be tentatively proposed as a more reliable and
sensitive method to measure the cognitive load, even in cases of
overload or underload.
To compare our classification results and computational load
with previous EEG-based studies, we replicated the classifica-
tion methodology used in [3], which reported 99% classification
accuracy for two load levels and 88% for four load levels (for
eight subjects). Using the same feature set (power), channels, and
window-length band, we achieved 83% classification accuracy
for seven load levels in the delta frequency, on our database.
Using the proposed wavelet features here, classification accuracy
of 98% for seven load levels across twelve subjects was achieved,
similarly to results based on our wavelet-entropy features in [31].
However, the two new and features here, gave
a slight classification performance improvement over
feature, which is the best performing entropy-based feature in
[31]. On the other hand, the feature has a shorter running
time and can be calculated faster than , as it follows. In
terms of the computational load, we compared the running time
of the two algorithms implemented in Matlab (R2010b) on a PC
(with Intel processor of core (TM)2 DUO CPU E8400@3 GHz).
The results show the running time of 66.46 and 46.12 seconds
for 65 seconds of data (for the selected channels, across twelve
subjects) with the proposed and features,
compared to 76.48 seconds for the feature in proposed in [3].
This indicates our proposed system outperforms the existing
systems and can be used for near real-time processing of EEG
signals for memory load classification.
As for other physiological measures, the average classifica-
tion accuracy of 57% from pupil diameter and 53% from GSR
measures were obtained for two load levels (across twenty sub-
jects) in [17]. We also evaluated the EEG signal stability over
time for two subjects and showed they are reproducible not only
after 6 months but also for high load levels, which is again a
promising preliminary result. Other studies to date have already
indicated that cognitive EEG is reproducible from one day to 18
weeks during a resting state or low cognitive load tasks [58].
We acknowledge a limitation in the above experimental work
that the order of the task presentation for the different task dif-
ficulty levels was not randomized for the test session. How-
ever, we repeated the recordings with randomized order at a
later stage for two subjects (recorded during the retest). The
extracted features in most of the channels of interest showed
the same decreasing trend and returned similar statistical re-
sults. For instance, for the entropy of the coefficients extracted
from Ch4 (F3) between L1 (L7) from the test session and L1
(L7) from the randomized order session, we have calculated
, using ANOVA test, indicating that the
two groups were taken from identical populations, across two
recording sessions. Therefore, we do not expect that the con-
sistent trends are due to the nonrandomized order of stimulus
presentation. It has been suggested previously that randomizing
the order of task presentation may reduce response order effect
but does not remove it entirely [59].
We note the other limitations of this experiment, including
the controlled conditions subjects were placed under while per-
forming the task (e.g., to sit still and refrain from excessive body
movement and blink). The setup was also lengthy and may have
induced some physical fatigue in the subjects prior to the start of
the experiment, however, we tried to minimize the fatigue im-
posed on subjects during the task recording by keeping the tasks
as short as possible and giving short breaks, after each task level.
We also acknowledge that this method should be validated on
a larger database with more subjects, to be able to generalize
the method as a subject-independent technique. Even though
the subjects were chosen to have a similar education level, there
may be some individual differences, such as better calculation
skills or motivation to complete the task, that may have affected
the performance or self-ratings.
VI. CONCLUSION
This paper demonstrates the promising capability of mul-
tiresolution features extracted from the EEG signals in discrim-
inating close levels of cognitive load imposed in an arithmetic
task. This discrimination was also demonstrated statistically
by means of Kruskal–Wallis and Friedman tests, for the se-
lected EEG channels, across twelve subjects. This is while the
Friedman ranking was neither consistent nor significant for
subjective self-rating. The results in Section IV overall suggest
EEG as a highly promising method for measuring cognitive
load to show an improvement over subjective self-ratings in
terms of statistical significance and classification accuracy.
The proposed cognitive load classification methodology,
which uses the entropy, energy and standard deviation of the
wavelet coefficients, outperforms self-ratings and existing
spectral-based features and achieves a very high detection
accuracy of 98% in discriminating seven load levels, across
twelve subjects studied (compared to 31% classification ac-
curacy of self-rating). In an informal experiment conducted,
we have extracted these features from artifact-affected data,
and not observed any degradation in classification accuracy.
This strongly suggests that the cognitive workload approach
described herein is applicable in near real-time, without any
manual artifact removal. The reliability and reproducibility of
ZARJAM et al.: BEYOND SUBJECTIVE SELF-RATING: EEG SIGNAL CLASSIFICATION OF COGNITIVE WORKLOAD
the reported results were also confirmed through the retest after
a six-month interval.
Future work includes investigating other features in the given
task load discrimination, collection of a bigger database, and in-
vestigating the applicability the proposed features for other cog-
nitive tasks. There is also scope for introducing further signal
preprocessing, towards improving the classification rate partic-
ularly for less controlled measurement environments.
REFERENCES
[1] O. T. Zander and C. Kothe, “Towards passive brain–computer inter-
faces: Applying brain–computer interface technology to human–ma-
chine systems in general,” J. Neural Eng., vol. 8, no. 2, pp. 1–5, 2011.
[2] P. Antonenko, F. Paas, R. Grabner, and T. van Gog, “Using electroen-
cephalography to measure cognitive load,” Edu. Psychol. Rev., vol. 22,
no. 4, pp. 425–438, 2010.
[3] D. Grimes, D. S. Tan, S. E. Hudson, P. Shenoy, and R. Rao, “Feasibility
and pragmatics of classifying working memory load with an electroen-
cephalograph,” in Proc. 26th SIGCHI Conf., 2008, pp. 835–844.
[4] F. Paas, J. E. Tuovinen, H. Tabbers, and P. W. M. Van Gerven, “Cog-
nitive load measurement as a means to advance cognitive load theory,”
Edu. Psychol., vol. 38, no. 1, pp. 63–71, 2003.
[5] A. Baddeley, “Recent developments in working memory,” Current
Opinion Neurobiol., vol. 8, no. 2, pp. 234–238, 1998.
[6] P. Sauseng, W. Klimesch, M. Schabus, and M. Dopplmayr, “Fronto-
parietal EEG coherence in theta and upper alpha reflect central execu-
tive functions of working memory,” Int. J. Psychophysiol., vol. 57, no.
2, pp. 97–103, 2005.
[7] W. Klimesch, M. Doppelmayr, and S. Dopplmayr, “Upper alpha ERD
and absolute power: Their meaning for memory performance,” in
Progress in Brain Research, N. Christa and K. Wolfgang, Eds. Am-
sterdam, The Netherlands: Elsevier, 2006, pp. 151–165.
[8] F. G. W. C. Paas, J. J. G. Van Merrienboer, and J. J. Adam, “Measur-
ment of cognitive load in instructional research,” Percept. Motor Skills,
vol. 79, no. 1, pp. 419–430, 1994.
[9] N. Meshkati, P. A. Hancock, M. Rahimi, and S. M. Dawes, “Tech-
niques in mental workload assessment,” in Evaluation of Human Work:
A Practical Ergonomics Methodology, J. R. W. E. N. Corlett, Ed., 2nd
Ed. ed. London, U.K.: Taylor & Francis, 1995, pp. 749–782.
[10] G. F. Wilson and F. T. Eggemeier, “Psychophysiological assessment
of workload in multi-task environments: Multiple-task performance,”
Multiple-task performance book, books.google.com, pp. 329–360,
1991.
[11] B. Yin and F. Chen, “Towards automatic cognitive load measurement
from speech analysis,” Lecture Notes Comput. Sci., pp. 1011–1020,
2007.
[12] N. Ruiz, R. Taib, Y. Shi, E. Choi, and F. Chen, “Using pen input fea-
tures as indices of cognitive load,” in Proc. 9th Int. ACM Conf., 2007,
pp. 315–318.
[13] F. Chen and Ruiz et al., “Multimodal behaviour and interaction as in-
dicators of cognitive load,” Trans. Intell. Interact. Syst., vol. 2, no. 4,
2012, Article 22.
[14] F. Paas and J. Van Merriënboer, “Instructional control of cognitive load
in the training of complex cognitive tasks,” Edu. Psychol. Rev., vol. 6,
no. 4, pp. 351–371, 1994.
[15] P. W. M. Van Gerven, F. Paas, J. J. G. Van Merriënboer, and H. G.
Schmidt, “Memory load and the cognitive pupillary response in aging,”
Psychophysiology, vol. 41, no. 2, pp. 167–174, 2004.
[16] Y. Shi, N. Ruiz, R. Taib, E. Choi, and F. Chen, “Galvanic skin response
(GSR) as an index of cognitive load,” in Proc. CHI ’ 2007 Conf., 2007,
pp. 2651–2656.
[17] E. Haapalainen, S. Kim, J. F. Forlizzi, and A. K. Dey, “Psycho-phys-
iological measures for assessing cognitive load,” in Proc. 12th ACM
Conf. , 2010, pp. 301–310.
[18] C. Berka and D. J. Levendowski et al., “Real-time analysis of EEG
indexes of alertness, cognition, and memory acquired with a wireless
EEG headset,” Int. J. Human-Comput. Interact., vol. 17, no. 2, pp.
151–170, 2004.
[19] A. Gevins, M. E. Smith, L. McEvoy, and D. Yu, “High-resolution EEG
mapping of cortical activation related to working memory: Effects of
task difficulty, type of processing, and practice,” Cereb. Cortex, vol. 7,
no. 4, pp. 374–385, 1997.
309
[20] G. Pfurtscheller and F. H. Lopes da Silva, “Event-related EEG/MEG
synchronization and desynchronization: Basic principles,” Clinical
Neurophysiol., vol. 110, no. 11, pp. 1842–1857, 1999.
[21] W. Klimesch, “EEG alpha and theta oscillations reflect cognitive and
memory performance: A review and analysis,” Brain Res. Rev., vol.
29, no. 2–3, pp. 169–195, 1999.
[22] C. Neuper, R. H. Grabner, A. Fink, and A. C. Neubauer, “Long-term
stability and consistency of EEG event-related (de-)synchronization
across different cognitive tasks,” Clinical Neurophysiol., vol. 116, no.
7, pp. 1681–1694, 2005.
[23] W. Klimesch, “Memory processes, brain oscillations and EEG syn-
chronization,” Int. J. Psychophysiol., vol. 24, no. 1–2, pp. 61–100,
1996.
[24] A. Gevins, M. E. Smith, H. Leong, L. McEvoy, S. Whitfield, R. Du, and
G. Rush, “Monitoring working memory load during computer-based
tasks with EEG pattern recognition methods,” Human Factors, vol. 40,
no. 1, pp. 79–91, 1998.
[25] A. Stipacek, R. H. Grabner, C. Neuper, A. Fink, and A. C. Neubauer,
“Sensitivity of human EEG alpha band desynchronization to different
working memory components and increasing levels of memory load,”
Neurosci. Lett., vol. 353, no. 3, pp. 193–196, 2003.
[26] C. Berka and D. J. Levendowski et al., “EEG correlates of task en-
gagement and mental workload in vigilance, learning, and memory
tasks,” Aviation, Space, Environ. Med., vol. 78, no. Supplement 1, pp.
B231–B244, 2007.
[27] A. Murata, “An attempt to evaluate mental workload using wavelet
transform of EEG,” Human Factors: J. Human Factors Ergon. Soc.,
vol. 47, no. 3, pp. 498–508, 2005.
[28] O. Jensen and C. D. Tesche, “Frontal theta activity in humans increases
with memory load in a working memory task,” Eur. J. Neurosci., vol.
15, no. 8, pp. 1395–1399, 2002.
[29] A. Bashashati, M. Fatourechi, K. R. Ward, and G. E. Birch, “A survey
of signal processing algorithms in brain–computer interfaces based on
electrical brain signals,” J. Neural Eng., vol. 4, pp. R32–R57, 2007.
[30] H. Nai-Jen and R. Palaniappan, “Classification of mental tasks using
fixed and adaptive autoregressive models of EEG signals,” in Proc. 2nd
Neural EMBS Conf., 2005, pp. 633–636.
[31] P. Zarjam, J. Epps, F. Chen, and N. H. Lovell, “Estimating cognitive
workload using wavelet entropy-based features during an arithmetic
task,” Comput. Biol. Med., vol. 43, no. 12, pp. 2186–2195, 2013.
[32] L. K. Mc-Evoy, M. E. Smith, and A. Gevins, “Test–retest reliability of
cognitive EEG,” Clinical Neurophysiol., vol. 111, no. 3, pp. 457–463,
2000.
[33] A. FÜrst and G. Hitch, “Separate roles for executive and phonological
components of working memory in mental arithmetic,” Memory Cogn.,
vol. 28, no. 5, pp. 774–782, 2000.
[34] M. Mc-Closkey, “Cognitive mechanisms in numerical processing: Ev-
idence from acquired dyscalculia,” Cognition, vol. 44, no. 1–2, pp.
107–157, 1992.
[35] S. Luck, “Setting up an ERP Lab,” in An Introduction to the Event-
Related Potential Technique (Cognitive Neuroscience). Cambridge,
MA, USA: MIT Press, 2005, pp. 305–340.
[36] A. Subasi, “Automatic recognition of alertness level from EEG by
using neural network and wavelet coefficients,” Expert Syst. Appl., vol.
28, no. 4, pp. 701–711, 2005.
[37] O. A. Rosso, S. Blanco, J. Yordanova, V. Kolev, A. Figliola, M. Schür-
mann, and E. Başar, “Wavelet entropy: A new tool for analysis of short
duration brain electrical signals,” J. Neurosci. Methods, vol. 105, no.
1, pp. 65–75, 2001.
[38] I. Guler and E. D. Ubeyli, “Multiclass support vector machines for
EEG-signals classification,” IEEE Trans. Inf. Technol. Biomed., vol.
11, no. 2, pp. 117–126, Nov. 2007.
[39] M. Akay, Time Frequency and Wavelets in Biomedical Signal Pro-
cessing. New York, NY, USA: Wiley-IEEE Press, 1998.
[40] P. Zarjam, J. Epps, and F. Chen, “Characterizing working memory load
using EEG delta activity,” in Proc. 19th EUSIPCO Conf., 2011, pp.
1554–1558.
[41] P. Zarjam, J. Epps, and F. Chen, “Spectral EEG features for evaluating
cognitive load,” in Proc. 33rd EMBS Conf., 2011, pp. 3841–3844.
[42] P. Zarjam, J. Epps, and F. Chen, “Evaluation of working memory load
using EEG signals,” in Proc. 2nd APSIPA Conf., 2011, pp. 715–719.
[43] D. C. Howell, Statistical Methods for Psychology. Belmont, CA,
USA: Thomson Higher Education, 2007, pp. 659–661.
[44] A. Fong, C. Sibley, A. Cole, C. Baldwin, and J. Coyne, “A compar-
ison of artificial neural networks, logistic regressions, and classification
trees for modeling mental workload in real-time,” in Proc. Human Fac-
tors Ergon. Soc. Annu. Meeting, 2010, vol. 54, no. 19, pp. 1709–1712.
310 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT, VOL. 7, NO. 4, DECEMBER 2015
[45] F. Lotte, M. Congedo, A. Léuyer, and B. Arnaldi, “A review of classifi-
cation algorithms for EEG-based brain–computer interfaces,” J. Neural
Eng., vol. 4, pp. R1–R13, 2007.
[46] L. Pezard, J. Martinerie, F. Breton, J. C. Bourzeix, and B. Renault,
“Non-linear forecasting measurements of multichannel EEG dy-
namics,” Electroencephalography Clinical Neurophysiol., vol. 91, no.
5, pp. 383–391, 1994.
[47] P. Zarjam, J. Epps, N. H. Lovell, and F. Chen, “Characterization of
memory load in an arithmetic task using non-linear analysis of EEG
signals,” in Proc. 34rd EMBS Conf., 2012, pp. 3519–3522.
[48] K. Sasaki, T. Tsujimoto, A. Nambu, R. Matsuzaki, and S. Kyuhou,
“Dynamic activities of the frontal association cortex in calculating and
thinking,” Neurosci. Res., vol. 19, no. 2, pp. 229–233, 1994.
[49] H. Kondo, M. Morishita, N. Osaka, M. Osaka, H. Fukuyama, and H.
Shibasaki, “Functional roles of the cingulo-frontal network in perfor-
mance on working memory,” NeuroImage, vol. 21, no. 1, pp. 2–14,
2004.
[50] N. Hazarika, J. Z. Chen, A. Tsoi, and A. Sergejew, “Classification of
EEG signals using the wavelet transform,” Signal Process., vol. 59, no.
1, pp. 61–72, 1997.
[51] E. D. Übeyli, “Combined neural network model employing wavelet
coefficients for EEG signals classification,” Dig. Signal Process., vol.
19, no. 2, pp. 297–308, 2009.
[52] J. Onton, A. Delorme, and S. Makeig, “Frontal midline EEG dynamics
during working memory,” NeuroImage, vol. 27, no. 2, pp. 341–356,
2005.
[53] T. Harmony et al., “EEG delta activity: An indicator of attention to
internal processing during performance of mental tasks,” Int. J. Psy-
chophysiol., vol. 24, no. 1–2, pp. 161–171, 1996.
[54] T. Fernández et al., “EEG activation patterns during the performance
of tasks involving different components of mental calculation,”
Electroencephalography Clinical Neurophysiol., vol. 94, no. 3, pp.
175–182, 1995.
[55] M. A. Vidulich and C. D. Wickens, “Causes of dissociation between sub-
jective workload measures and performance: Caveats for the use of sub-
jective assessments,” Appl. Ergon., vol. 17, no. 4, pp. 291–296, 1986.
[56] Y.-Y. Yeh and C. D. Wickens, “Dissociation of performance and sub-
jective measures of workload,” Human Factors: J. Human Factors
Ergon. Soc., vol. 30, no. 1, pp. 111–120, 1988.
[57] E. N. Wiebe, E. Roberts, and T. S. Behrend, “An examination of
two mental workload measurement approaches to understanding
multimedia learning,” Comput. Human Behav., vol. 26, no. 3, pp.
474–481, 2010.
[58] L. M. Williams, E. Simms, C. R. Clark, R. H. Paul, D. Rowe, and E.
Gordon, “The test-retest reliability of a standardized neurocognitive
and neurophysiological test battery: “Neuromaker,” Int. J. Neurosci.,
vol. 115, no. 12, pp. 1605–1630, 2005.
[59] J. A. Krosnick and D. F. Alwin, “An evaluation of a cognitive theory
of response-order effects in survey measurement,” Public Opinion
Quart., vol. 51, no. 2, pp. 201–219, 1987.
Pega Zarjam received the master’s degree from Queensland University of
Technology, Brisbane, Australia, and the Ph.D. degree in biomedical signal
processing from the University of New South Wales, Sydney, Australia,
respectively.
Her research interests include biomedical/physiological signal processing,
time-scale/time-frequency signal processing, and pattern recognition.
Julien Epps (M’97) received the B.E. and Ph.D.
degrees from the University of New South Wales,
Sydney, Australia, in 1997 and 2001, respectively.
After an appointment as a Postdoctoral Fellow at
the University of New South Wales, he worked on
speech recognition and speech processing research
first as a Senior Research Engineer at Motorola Labs
and then as a Senior Researcher at National ICT Aus-
tralia. He joined the UNSW School of Electrical En-
gineering and Telecommunications as a Senior Lec-
turer in 2007, and is currently an Associate Professor.
His research interests include characterization, modeling, and classification of
mental state from behavioral and physiological signals. He has authored or coau-
thored around 180 publications.
Dr. Epps is currently serving as an Associate Editor for the IEEE
TRANSACTIONS ON AFFECTIVE COMPUTING, as well as regularly serving as a
Reviewer or Technical Program Committee Member for several IEEE Journals
and numerous conferences.
Nigel Lovell (M’91–SM’99–F’11) received a B.E.
degree (Hons) in electrical engineering, in 1983,
and the Ph.D. degree in biomedical engineering, in
1991, both from the University of New South Wales,
Sydney, Australia.
He is currently at the Graduate School of Biomed-
ical Engineering University of New South Wales
(UNSW), Sydney, Australia, where he holds a
position of Scientia Professor. He has authored 200+
refereed journals and 270+ conference proceedings,
and been awarded over $79 million in R&D and in-
frastructure funding. His research work has covered areas of expertise ranging
from cardiac modeling, telehealth technologies, biological signal processing,
and visual prosthesis design. Through a spin-out company from UNSW he
has commercialized a range of telehealth technologies for managing chronic
disease and falls in the older population. He is also one of the key researchers
leading an R&D program to develop an Australian bionic eye.
Dr. Lovell is a Fellow of five learned societies throughout the world: ATSE,
Engineers Australia, IEEE, FIP, and AIMBE.