Accepted Manuscript

Correlation between Videogame Mechanics and Executive Functions through

EEG analysis
Tania Mondjar, Ramn Hervs, Esperanza Johnson, Carlos Gutierrez, Jos
Miguel Latorre
PII:
DOI:
Reference:

S1532-0464(16)30083-1
http://dx.doi.org/10.1016/j.jbi.2016.08.006
YJBIN 2614

To appear in:

Journal of Biomedical Informatics

Received Date:
Revised Date:
Accepted Date:

2 March 2016
4 August 2016
5 August 2016

Please cite this article as: Mondjar, T., Hervs, R., Johnson, E., Gutierrez, C., Latorre, J.M., Correlation between
Videogame Mechanics and Executive Functions through EEG analysis, Journal of Biomedical Informatics (2016),
doi: http://dx.doi.org/10.1016/j.jbi.2016.08.006

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers
we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and
review of the resulting proof before it is published in its final form. Please note that during the production process
errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Correlation between Videogame Mechanics and Executive

Functions through EEG analysis
1,2

1a

1a

1a

1b

Tania Mondjar , Ramn Hervs , Esperanza Johnson , Carlos Gutierrez , Jos Miguel Latorre
1

University of Castilla-La Mancha

Department of Technologies and Information Systems
Paseo de la Universidad 4, Ciudad Real, Spain
b
Department of Pshychology
Almansa, 14, Albacete, Spain

eSmile, Psychology for Children & Adolescents.

Calle Toledo 79,1E. Ciudad Real, Spain.

Abstract. This paper addresses a different point of view of videogames, specifically serious
games for health. This paper contributes to that area with a multidisciplinary perspective focus on
neurosciences and computation. The experiment population has been pre-adolescents between the
ages of 8 and 12 without any cognitive issues. The experiment consisted in users playing videogames as well as performing traditional psychological assessments; during these tasks the frontal
brain activity was evaluated. The main goal was to analyse how the frontal lobe of the brain (executive function) works in terms of prominent cognitive skills during five types of game mechanics widely used in commercial videogames. The analysis was made by collecting brain signals
during the two phases of the experiment, where the signals were analysed with an electroencephalogram neuroheadset. The validated hypotheses were whether videogames can develop executive
functioning and if it was possible to identify which kind of cognitive skills are developed during
each kind of typical videogame mechanic. The results contribute to the design of serious games
for health purposes on a conceptual level, particularly in support of the diagnosis and treatment of
cognitive-related pathologies.
Highlights:
- Typical videogame mechanics exercise the prefrontal brain area, making it possible to improve
cognitive skills through the design of serious games.
- There are particular videogame mechanics with strong correlation to specific cognitive processes,
such as inhibition, working memory, and shifting.
- The combination of neurosciences and computation makes possible the improvement of diagnosis and treatment of cognitive issues.
Keywords: serious games, pervasive health, health games, brain-computer interaction, executive
functions, computational neurosciences, cognitive rehabilitation, electroencephalogram.

Introduction

Executive function (EF) comprises a family of mental processes associated with the functions of the
prefrontal cortex [1]. More specifically, it refers to high-level cognitive processes oriented towards
reactive inhibition and the regulation of goal achievement behaviour [2]. Three main components have
been identified: inhibition (suppression of affective driven behaviours); working memory (to hold
information active in mind and to mentally work with that information as a platform for guiding our
behaviour); and shifting (switching flexibly between tasks or mental sets) [3]. With respect to other
domains, the EFs have been considered essential for physical and mental health, academic achievement and cognitive, social, and psychological development [4, 5, 6]. EF deficits have been associated
with neurodevelopmental disorders [7, 8, 9] and different types of psychopathology [10, 11]
EFs are directly related with brain development. This development happens in an increasing way,
and progressively. These changes are produced, influenced by the experiences of people, which give
place to the maturity of the prefrontal cortex in terms of the interaction with the environment. In this
way, the neural networks that sustain EFs are moulded [12]. This development of EFs reaches its
adulthood average at around 12 years old [13].
The activities that we develop in our day-to-day experiences influence our cognitive development.
Considering this and analysing the activities that children dedicate a significant amount of time to, it is
possible to establish the connection of videogames and cognitive training and rehabilitation. Currently
videogames are within everyones reach, offering a multitude of gaming mechanics, platforms, and the
ability to be present in most technological devices. The use of this technology has surpassed the limits
of its main purpose, which is entertainment, to offer other goals. Authors such as Piaget and Vigosky
stated that childrens games are essential to cognitive processes and personal-emotional development.
The cognitive skills above are tightly related to traditional games, and can be extrapolated to videogames [14].
This manuscript contributes to the successful design of serious games for health. These kinds of
games aim to achieve goals beyond just entertaining: simulators, virtual worlds, and interactive environments bring amazing possibilities in different areas. Particularly, there are serious games focused
on health, e.g., games applied to learning about living a healthy life [15], exergames to motivate
healthy behaviours [16] and virtual worlds for psychotherapy [17]. Expanding this concept, this paper
explores the use of serious games to cognitive development and rehabilitation, an innovative approach
of using health games. These kinds of games, with an appropriate design, can be used by individuals
with psychological impairments during rehabilitation sessions [18], in particular, they can help to
evaluate and improve processes such as the attention, basic perception, and capacity of extracting patterns [19], among others.
In more detail, this paper aims to carry out an analysis of brain activity during the psychological
evaluation of EF and while users are playing videogames. This paper is not focused on any particular
genre of videogames; we have identified mechanics that can be present in diverse genres such as ac-

tion games, adventure games, role-playing games, or strategy games. Regarding the substantial
amount of mechanics that can be included in a videogame, we have focused our study on those that
require cognitive processes related to EFs (at least, in theory, this is one of the hypothesis of this
work). For example, we are interested in the mechanics that require working memory, planning, attention, and inhibitory response. Additionally, we have avoided mechanics that require some cognitive
processes such as long-term memory, previous knowledge, or language skills. Once the videogame
mechanics to be included in this study were identified, several specific videogames were designed
based on the cognitive abilities required in each one. Thus, the contribution of this paper is to explore
the correlation between several game mechanics and EFs. This correlation helps in the development of
videogames that aim to evaluate or improve cognitive skills.
The paper is organised as follows: Section 2 describes the motivation for this work; and Section 3
presents the overall experiment that has been conducted, including the target population, the method
where we explain the starting research question, the work hypothesis and the different experiment
stages, the used materials, and the experiment results. This third section is the core of the paper. Section 4 reviews the background to this work and related proposals, highlighting the differences and
similarities between these works and our proposal. Finally, Section 5 discusses the lessons learned
from this work and suggests directions for future research, and Section 6 concludes the paper.

Justification

The evolutionary development and the activities that we perform concern the cognitive maturity of
our brains. From birth and thanks to neural plasticity, the brain generates, renews, and models the
synaptic transmissions between brain cells with the end goal of making a more extensive and expert
system based on experiences. For this reason, the exercising of concrete cognitive abilities since
childhood and adolescence helps cerebral neurotransmission improve their role in behaviour.
In particular, people with cognitive impairments suffer from significant limitations in their intellectual functions, such as reasoning, learning, and problem solving. These limitations affect their conceptual, social, and practical skills, which can have a substantial emotional and behavioral impact on their
lives. Cognitive impairments lead to difficulties in performing everyday activities, and these difficulties are even harder when appearing in children of developmental age.
Methods to mitigate and rehabilitate cognitive deficits are of great importance. Rehabilitation typically consists of a set of interventions that help to increase the functional capabilities of the individual.
The interventions can be focused on either functionality (enhancement of cognitive skills) or processes (improvement of functioning). Rehabilitation strategies can be divided into three main categories:
restitution, or stimulation of the affected neuropsychological connections; compensation, or enhancement of functioning through healthy neuropsychological systems; and substitution, which employs
external procedures that help the affected individual at the practical level (external aid).

Our work points to interventions focused on functionality and processes, and can improve rehabilitation based on restitution strategies. Thus, the objective of this work is to offer an alternative to encourage cognitive training and rehabilitation, improving connection on a neurological level, on top of
facilitating neuropsychological evaluations on child and adolescent populations with or without associated problems (i.e., disability). In this way, we combine a very entertaining and accepted activity by
this collective with a double end: entertainment at the same time that gamers improve their cognitive
and personal development.
The fundamental of this work is the measurement of brain waves activation through an electroencephalogram (EEG) to relate the psychological components with physiological measurements. This
practice is widely used in psychophysiology [20] where brain activity is observed during different
situations to obtain inferences of the correlations between activities. This type of correlation does not
have prototypical models that are generally accepted; therefore, a simplified comparative protocol has
been established. In our case, the activities to compare are certain standardized psychological assessments adapted to the target population, and in particular, videogames. This contribution is the innovative proposal of this work. There are other proposals in this area (see Section 4) but they do not generalize their results to the EF in their entirety by establishing concrete activation patterns and, consequently, they cannot be directly applied to the successful development of serious games for cognitive
training and rehabilitation.

Experiment

3.1

Population

The population of this experiment was n=12, in which there are 6 boys and 6 girls, ages between 8
and 12 years (therefore belonging to a cohort of preadolescents), with none of them having been diagnosed with any cognitive issue. The collected data during this experiment as well as the scope and
goals of this research was clearly explained to the subjects and their parents, who signed the information sheet and consent form to obey mandatory ethical principles.
3.2

Method

The experiment gathered evidences in terms of EEG data in an empirical manner, with the data analysed qualitatively and quantitatively. The question that guides this experiment is: How can action
videogames be designed to develop cognitive skills related to executive functioning? This question
made us propose two hypotheses to be testes through this experiment:

Hypothesis 1: Action videogames develop the cognitive processes related to executive

functioning.

Hypothesis 2: Typical mechanics involved in action videogames stimulate the same areas as psychological assessments that evaluate executive functioning, making possible
the correlation between videogame mechanics and cognitive skills.

The experiment is divided in two stages, which are psychological assessment-based evaluation and
a videogame-based evaluation.
The protocol of the experiments is summarized as follows: (1) participants were informed about the
EEG device and the data it would collect; (2) they were required to wear the headset, ensuring their
comfort and trust, (3) the psychologist explained the data that was seen on the screen, where participants could observe their brain activity; (4) the psychologist gave instructions about the tasks that
were to be performed, ensuring the participants that their abilities were not being tested, adding that it
is impossible to read minds with the EEG device (a concern among many children); and (5) participants performed the tasks with no interruptions by the researchers observation or the EEG monitoring. Both phases of the experiment followed this protocol, and the participants wore the EEG headset
for approximately 40 minutes. Once the EEG data was acquired, we observed the individual graph of
the group of electrodes situated in the frontal part of the brain (channels) where we can differentiate
the greater use of this part of the brain in terms of its graphical representation. On the other hand, we
observe the activation of different band frequency or emitted waves by the brain (theta, beta, alpha,
and delta) and the area in which they are more active. These measurements were done while the participants performed the psychological assessments, and while they were playing the videogames.
Phase one of the experiment aims to define a cognitive model, which consisted of neurological patterns of EFs [21], through psychological assessments of EFs in children; in particular, there were three
tests called trails, washers, and interference [22]. Section 3.3.1 describes these assessments in detail.
During the second phase, participants used videogames while their EEG data was recorded. This
phase followed the same protocol except for the directions, which were unnecessary to guide the activities due to the natural skill of children in general when it comes to understanding and using videogames.
Both phases were recorded with a video camera (preserving participant anonymity), and the EEG
dataset was generated and labelled for the subsequent analysis. After the collection of all the data, we
proceeded to the qualitative and quantitative analysis to support or dismiss the declared hypotheses,
and answer the main research question of this work. Two perspectives were followed when the data
analysis was performed: (a) Observational method based on the psychologists expertise in regards to
EEG data to identify segments of data and characteristic cognitive patterns, and (b) Signal processing
algorithms to quantify the similarity of the identified segments.

3.3

Materials

For the development of this work, we used several computational and neuroscience tools and devices.
In the following subsections, the main materials used in this research are explained: three psychological assessments, EEG analysis devices and tools, and the developed videogames.
3.3.1. Psychological assessments to evaluate executive functioning
The assessments that have been used are three neuropsychological tests that measure, in a reliable
way, the abilities that are grouped within EFs. One of the tests used is an adaptation of the Trail Making Test (TMT) because of its acceptation. This test was used in army experiments and was included
in the Halsted Reitan battery of Reitan and Wolfson. It remains one of the most frequently used neuropsychological tests in research and clinical practice [23]. For this project we have used an adaptation for children included in the battery of neuropsychological tests, ENFEN [22], called the Trails
test, because of its reduced complexity and to lessen the main weak points of the TMT, such as the
age slant and the influence of academic level [24][25]. In this case, the test is centered on abilities like
selective attention, perceptual speed, and working memory.
Another assessment for this experiment has been the interference test. It is an adaptation of the
well-known test of Stroop (1935), which evaluates the capacity to inhibit a typical answer if we
change the demands. This test is used today to detect brain damage [26]. For this reason, its adaptation has been used to observe activity in the frontal part of the brain, where inhibition processes take
place.
Finally, we have worked with the Washers test, an adaptation of the Tower of Hanoi (TOH). This
test lies in manipulating rings over three fixed bases to construct the previously indicated patterns in
the least time possible and with the least amount of moves. The Tower of Hanoi problem has become
a tool in cognitive psychology to understand planning both from an information-processing perspective [27] and a cognitive neuroscience perspective. In this test we evaluate planning, guidance, control, and problem solving, which have a direct relation in behavior execution. Its choice is also influenced because it is not correlated with an intellectual quotient. A summary of related cognitive skills
with each psychological assessment is shown in Table 1.
Table 1. Cognitive skills related to psychological tests
Assessment

EF Component

Cognitive Skills

Trails (TMT)

Working Memory

Decision making, Problem solving, Selective attention, Perceptual speed, Anticipation and foresight

Washers (TOH)

Shifting

Planning and sequencing, Spatial orientation, Abstraction capacity, Decomposition problems

Interference
(Stroop)

Inhibition

Selective attention, Sustained attention, Ability to inhibit, Resistance to interference, Classify

3.3.2. EEG analysis

We used the Emotiv EPOC+ neuroheadset1 to collect the EEG data. This device has 14 EEG channels
and two references for positioning and accurate spatial resolution. These channels have eight frontal
electrodes (AF3, F7, F8 and FC5 on the left hemisphere, and FC6, F4, F8 and AF4 on the right hemisphere), two temporal electrodes (T7 and T8), two occipital electrodes (O1 and O2) and two parietal
electrodes (P7 and P8). The headset uses a sequential sampling method at a rate of 128 samples per
second. To analyse the EEG dataset, we used the Xavier TechBench Software
ize the signals, and MatLab R2015a

TM

TM

to collect and visual-

to develop the algorithms for signal processing and metrics. We

also developed a software using C# and the Emotiv EPOC+ API to improve the gathering and conversion needed for the dataset format. The dataset was stored in one file per subject. Table 2 includes the
description of the different fields into the data set. It is important to remark that the data used in the
analysis only took into account the brainwave signals of the 14 monitored channels. The order of the
channels in the data is F3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4.
Table 2. Dataset field description
Field

Description

Counter

Packet counter running from 0 to 128 (based on device sampling rate)

AF3 AF4

EEG brain signal channels (14 electrodes)

CQ_AF3CQ_AF4

Contact quality of each electrode (14 electrodes)

For this project we decided to use the Emotiv EPOC+ headset after comparing several devices created for this end (NeuroSky, Neuronetrix H1200, and actiCAP). This device allows for use during a
long time as a non-invasive method, and it is wireless and very light, which makes it adequate for
working with children. A few more strong points were its adaptation to the size of the head and its
ease of handling and usage. In practice, the EEG has some limitations as the EEG signals can be very
sensitive to noise and pick up unwanted artifacts [28] caused by many factors such as facial movements like blinking or mouth movements [29]. For this reason, that EEG Data is not directly analyzed
as a whole; less noisy and more significant frames are identified prior to the quantitative analysis.
In general, the Emotiv EPOC+ device is able to record EEG data in a satisfactory manner. There are
several studies that evidence the significant worse performance of this device compared with professional medical devices. However, some of the studies remark the validity of this device for noncritical proposes, such as for games [30]. The level of precision is not a strict requirement in our study
because it is not necessary to collect highly accurate brain activity. The experiment lies in comparing
brain activity during the performance of activities related to EFs and with the use of videogames. In
this case, the device performs with an acceptable level of accuracy. Moreover, the election of this device is also because it is easy to use with children during the experiment time and it has a rapid cali-

https://emotiv.com/epoc.php

bration that is required for laboratory studies. In order to ensure an optimal performance, it is needed
to place the reference sensors (rubber sensors) just behind each ear lobe and the 2 front sensors about
3 cm above the eyebrows. The device software automatically carries out the calibration process pressing and holding the reference sensors for about 5 seconds.
3.3.3. Developed videogames based on main game mechanics
Several students of the Faculty of Computer Science in Ciudad Real (University of Castilla-La Mancha) developed a set of simple videogames focused on specific action mechanics. Figure 1 shows a
picture of the conducted experiment including the materials involved.

Fig. 1. Performed experiment and materials

It is necessary to delimit the kind of games to include in this research due the unlimited variety of
videogames in existence. The chosen videogame category is the named action videogames, a type of
videogame that requires the ability of the user to face proposed challenges. This genre of videogames
can test out several user skills closely related to EFs. For this work, we have identified several typical
mechanics in commercial action videogames. In the following, we describe the identified mechanics,
including the five most representative ones. Theoretically, each of them requires different cognitive
processes and skills to be successfully overcome. This statement has been tested during the experiment. The chosen mechanics are explained below, including the expected related cognitive skills due
to the nature of each mechanic:

Accurate action: in this mechanic the player has to make a set of actions to continue with
the game in a precise and careful way. These actions are related with processes such as
concentration, attention, impulse control, and information comprehension among others.
(Examples of commercial games: Tomb Raider; Castlevania; Assassins Creed.)

Timely action: in this case the player performs an action in a defined moment that is determined by events in the game, where the timing of the action is the most important thing.
This mechanic is related to working memory, selective attention, decision-making, problem
solving, and perception. (Examples of commercial games: God of War; Heavenly Sword.)

Mimic sequence: in this mechanic, players have to reproduce a sequence of actions previously shown in the game (explicit or implicitly). The processes related to this are working
memory, focalized attention, and inhibition control. (Examples of commercial games:
Zelda; Fighting games.)

Pattern learning: the player has to learn a pattern in terms of events in the game. In this
case the player has to use processes such as selective attention, planning, inhibition control,
and spatial orientation to continue with the game. (Examples of commercial games: Thief;
Prince of Persia; Hitman.)

Logical puzzles: in this case the player has to understand a logical behavior of events in
the game to go forward. In this mechanic we use attention, working memory, the capacity
for abstraction, information processing, problem solving, or resistance to interference. (Examples of commercial games: Silent Hill; Resident Evil.)

Most commercial videogames blend all of those mechanics as well as having other mechanics related to adventure features (e.g., scenery exploration, object gathering, long-term planning, etc.). On
top of that, unlocking some of those mechanics would require the person to play for several hours.
Also, commercial games include elements such as music, sounds and animations that can be distractors and could divert attention or activate, by their own, some cognitive processes. For these reasons,
several specific games were developed to support this research, each of them focused on a particular
game mechanic and avoiding external distractors.
Figure 2 shows these games, from left to right and top to bottom, the descriptions are as follows:
Dreamskeeper has a mechanic consisting in moving the mouse to avoid obstacles. KittenQuest is a
classic running game, where the mechanic is to miss obstacles in their way in a timely manner.
CrazyFarms main mechanic is to show visual and sound sequences that have to be mimicked. Apis
Adventures is a plataform game in which the particular patterns that enemies repeat has to be identified to reach the next levels. Finally, Kunoichi has the user planning a set of basic actions in terms of
logical movements to exit the gate.

Fig. 2. Screenshots of DreamsKeeper (Accurate Actions), KittenQuest (Timely actions), CrazyFarm (Mimic
sequences), Apis Adventures (Learning patterns), and Kunoichi (Logical Puzzles).

3.4

Results

This section explains how observations and collected data were qualitatively and quantitatively analysed to obtain the associated results.
3.4.1. Observational analysis of data
The results were evaluated by the observational method focused on electric impulses in the brain that
represent an interface line emitted through the electrodes of the EMOTIV EPOC+ headset. With this
method, the electric brain activity patterns are interpreted by comparison of participants and activities.
The interpretation of the results was performed by visual analysis of the graphical representation of
EEG signal.
Firstly, the results of the participants show high activity in the electrodes AF3, F7, F8 and AF4, which
correspond to the frontal area of the prefrontal region. We can observe in Figure 3 that signals show
different levels of activation, and that, compared with the rest of the electrodes, parietal temporal and
occipital areas show lower activation levels. In this phase, we emphasize that the level of activation in
the frontal lobe matches the measure of the skills assessed in the previous phase of neuropsychological
assessments, which was expected as the tests were validated for such a purpose. Similarly we can see
high activation in the frontal area in the second phase of the evaluation (videogame phase), which
stand out against the middle to low activity levels in the rest of the brain electrodes. With this data we
can objectively affirm that during the assessments and the use of videogames, the brain uses the skills
located in the frontal area more significantly.

Fig. 3. Example of EEG data collected during the experiment.

To support these observations, the EEG signals visualized in the brain surface were analysed. Some
findings are explained in the following. Beta activity is most evident in the frontal cortex and is connected to some cognitive processes like decision-making, problem solving, and information processing [31]. Theta activity seems to be connected to creativity, memory recall, emotions, and sensations [32]. In this experiment, greater significance is given to the activation of theta waves when the
player is demanded to give a new answer (different mechanic) in the frontal activation and in a generalized way in most users. We can also observe the activation in terms of beta waves when remembering a necessary answer to continue in the game in the frontal area and in some point on an occipital
level. The general activation of the three types of waves (delta has no bearing as it is present during
sleep), particularly in beta waves, occurs in situations that provoke a change in concentration such as
during times of stress or frustration when failing. Thus, theta waves are related with the frontal area of
the brain in activating in situations of analysis of information, provoked by necessities in terms of the
selected mechanics.
As we can see in Figure 4, the area in red is mainly limited to the front surface of the head. Besides
the identified similarities in the use of the frontal lobe, some differences can be seen between the
hemispheres that point to a more emotional presence during the use of videogames.

Fig. 4. Activation brainwave on trails (left) and timely action (right)

Fig. 5. Example of correlation observed by the expert: EEG Signal activation in Washers test (right) and learning patterns mechanic (left).

3.4.2. Quantitative analysis of data

The EEG signals are non-Gaussian, non-stationary and non-linear in nature, and consequently it is
hard to analyse them in an automatic way. Usually, EEG signals are analysed through expert observation, who can distinguish normal and abnormal states (explained in section 3.4.1). Because observational results tend to be general, we have included a second test to quantify similarity levels between
EEG signals.

Fig. 6. Example of the anterior frontal electrode (AF3) with the participant 4 while doing activity Washers (Blue
line) compared to the five studied mechanics (respectively Red, Green, Yellow, Magenta, and Cyan lines). The
figure shows the consecutive signal processing phases: (1) identification of significant time frames; (2) filtering
through low-pass filter; (3) segmentation; (4) splitting; and (5) gathering of similarity variables (e1-gn for AF3
using Cross-Correlation Coefficient and e1-gn for AF3 using Dynamic Time Warping, for each segment of
EEG data).

Figure 6 shows the consecutive stages to compute the time frames of the EEG signals, which allows us to find similar cognitive patterns. From the observations of the expert, significant time frames
were identified and included in datasets, which usually lasted five seconds (640 sequential samples).
An example of significant frames is shown in Figure 5. The comparison was done with each assessment activity and each videogame, analysing each electrode separately. The signals were normalized
and then, a low-pass filter was applied. It is necessary to use a filter because the raw data of the EEG
is noise and redundant. Consequently, we have chosen the following smoothing function (Equation 1).
S(At) is the filtered EEG frame and At is the EEG raw frame output, which is acquired by the headset
at time t. Moreover, is a smoothing factor in the range from 0 to 1. The factor is critical for acquiring valid data to be analysed in the correlation process. The applied =0.15 was determined experimentally.

(1)

The softened signal was segmented to separate complete signal periods. Finally, these segments
were analysed using two kinds of similarity methods: a statistical method through cross-correlation
coefficient (CCC) and an algorithmic method named Dynamic Time Warping (DTW).
CCC is a method to determine the degree to which two signals or numerical series are correlated.
The standard formula for the CCC between two signals x and y is shown in Equation 2, with the time
stamp l, and where

and

are mean values and K+1 is the EEG frame length.

(2)

DTW computes the distance between two EEG frames () by finding the minimum path that will be
represented with a numerical value. In this experiment, the averaged Euclidean distance defines the
cost between two different points from the EEG frames that are being compared. We defined a DTW
window parameter of 50 that was selected by experimentation and based on our previous experience
in other applications using that algorithm [33-35].
By means of this procedure, we obtained similarity values to each pair assessment-videogame, for
each significant segment of each electrode. Table 3 shows the averaged similarities taking into account the twelve participants. The data shown in Table 3 has been scaled to the interval [0,1] using
min-max normalization both DTW results and CCC results separately, because these two methods
cannot be compared directly. The normalization helps us to identify those mechanic-test pairs most
correlated looking the smallest correlation values.
Table 3. Averaged similarity between assessments (e1-e3) and adventure-action videogames mechanics (g1-g5)
through min-max normalized CCC () and DTW ().
Videogames Mechanics
Accurate

Timely

Mimic

Learning

Logic

Action

Action

Sequence

Patterns

Puzzles

e-g1

e-g1

e-g2

e-g2

e-g3

e-g3

e-g4

e-g4

e-g5

e-g5

Trails

0,83

0,86

0,07

0,00

0,52

0,58

0,41

0,54

0,46

0,25

Washers

0,81

0,63

0,22

0,62

0,51

0,58

0,20

0,10

0,22

0,38

Interference

0,92

0,97

0,43

0,58

1,00

1,00

0,28

0,34

0,00

0,14

In order to have a better understanding these quantitative results, Figure 7 shows a graphical representation using radial charts. This figure again represents the correlation between each videogame mechanic and the three psychological assessments. It is important to remark that the plotted data is the

normalized correlation with the sign changed, just for visual representation purposes, easing the chart
understanding. We can see the higher correlation in the pairs: Trails-Timely Action, WashersLearning Patterns, and Interference-Logical Puzzles. However, results denote that the correlation is
not clearly one-to-one; each game mechanic has similarities with several assessments, and in consequence, they work several cognitive skills. For example, it is possible to establish less steady correlations in the pairs: Trails-Logical Puzzles, Washers-Logical Puzzles, and Interference-Learning patterns. Another interesting finding is that the mechanics of Logical Puzzles and Learning Patterns are
the more related with EF activities and, consequently, they are the most powerful mechanics in terms
of cognitive training and rehabilitation. On the other hand, Mimic Sequences and Accurate Actions
are the mechanics showing less correlation with EFs. Finally, focusing on the area of each assessment
in the chart that the assessment Washers is more related with the set of game mechanics followed by
Trials and Interference.

Fig. 7. Graphical representation of normalized correlation obtained from averaged CCC and DTW results that
compare psychological assessments Trails (TMT), Washers (ToH), and Interference (Stroop) with five game
mechanics.

Background and Related Work

Typically, serious games deal with integrating educational goals and game mechanics based on wellknown tests to enhance and generalize learning to become significant [36], as well as helping to discover and use information instead of just memorizing it [37]. In regards to the area of health, videogames are becoming a successful tool to teach and train areas to treat chronic diseases and disability.
One example is the videogame Re-Mission, which helps children to understand cancer by fighting
malignant cells [38]; another example is Insouline, which deals with the issue of controlling blood

sugar levels through metaphors in games for cases of children diagnosed with diabetes [39], and finally WiiPD that has several minigames to analyse motor tasks of people with dementia [40]. Related to
the area of psychology, the proposals range from treasure hunts to support cognitive-behavioural therapies [41], game systems for cognitive rehabilitation [42], videogames for the control of pain perception and management [43], and games with the goals of self-management of emotions (e.g., anger
control)[44].
The use of EEG during gaming is being analysed to see what happens in the brain or to offer
neurofeedback to users. There are studies related to cognitive abilities with existing games in the market such as those focused on attention using the Tetris game [45], or proposals centred on decision
making with The Sims 2 [46]. Other examples seek to improve or explore brain activity in certain disorders such as separation anxiety disorder with Coping Cat [47], Attention problems related to Autism [48], EEG analysis during resting condition with open eyes [49], and emotion identification [50].
Other studies center on offering feedback to the user such as Wang et al. [51], which propose a nonlinear fractal dimension based approach to neurofeedback implementation targeting EEG-based serious games designs, or Camerio et al. [52] who centered on cognitive rehabilitation.
With the awareness of the state of the art in the context of health games, the goal of this paper was
the identification of cognitive skills developed while using serious games based on action videogames.
Unlike some of the related work, which usually backed up psychological theories with an evaluation
through qualitative experiments, this proposal is supported by computational neurosciences, and it has
been validated by means of both a quantitative and a qualitative approach. Using EEG that allows the
analysis of active brain areas during cognitive exercises, both in psychological assessments and videogames, we performed the evaluation of cognitive skills. Some of the related work that shares points
with our proposal include the use of EEG and the support of serious games, but an important feature
of our work is the use of both, as well as working with a set of cognitive abilities that are tied to the
executive function, and analyzing general mechanics instead of focusing on particular commercial
videogames, which allows us to infer what kinds of videogame mechanics are useful to include in
application for diagnosis and improvement of cognitive skills.

Discussion and future work

This work contributes to the development of videogames with EFs diagnosis and training purposes
through the identification of videogame mechanics that involve brain activity patterns related to particular cognitive skills. To support the contribution in diagnosis and rehabilitation, the work leans on
evidences about the improvement of EF through the exercising of these mental processes [53]. The
strongest evidence for an activity improving childrens EFs exists for CogMed computerized training [54]. At any age across the life cycle EFs can be improved, including in the elderly and in infants.
Much, but not all, of the work on improving EFs in young adults has focused on computerized training. Exposure to bilingual input has been one of the foci, though not the only focus, of work on accel-

erating the development of EFs in infants [55]. Although the improvement of EF is solidly supported
in the literature, future work is needed that would include the long-term evaluation of the use of videogames, which also includes the studied mechanics to validate and quantify the improvement of EF.
In this sense, like Bisoglio [56] has said, it is necessary to attend to the individual differences that
constitute continued research to make it possible to check if the improvements in brain activity (particularly in EFs) are a consequence of playing videogames that are complementary to natural development.
Regarding the applied methods to process the brain activity, it is important to remark that EEG signal processing is a study field in progress due to the particular nature of these signals. In this work, we
firstly focused the signal processing on collecting statistical variables. An interesting alternative is to
apply advanced signal processing techniques that the literature has demonstrated as very valuable,
e.g., different kind of entropies, Hurst exponent and Higher Order Spectra [57]. However, even proving these methods to identify anomalies associated to pathologies in EEG signals, our goal is just
identifying similarities to correlate cognitive skills during activities of neuropsychological assessments and during the use of videogames. In this particular case we relied on the expert observation to
collect significant time frames of data, which enable the study of similarity level in short temporal
series, and we obtained relevant results through the CCC and DTW methods. However, it is not possible to distinguish which similarity value is more accurate. On one hand, CCC is better to compare a
signal in different scales. On the other hand, DTW allows comparing signals, which may vary over
time. For this reason, we took into account both similarity coefficients.
The developed experiment and the collected datasets could be used to aid the diagnosis and treatment related to EF issues The developed experiment and the collected datasets could be used to aid
the diagnosis and treatment related to EF issues through the game design. By using supervised learning techniques [58] such as Support Vector Machine (SVM) or Hidden Markov Models (HMM), it is
possible to identify normal and abnormal cognitive behaviour when using videogames. In that case, it
requires the identification of a productive and limited statistical feature set [59], being that one of the
further challenges. This is particularly the scope for future work to be developed by means of those
supervised learning techniques and by increasing the population of the research, including more participants with and without diagnosed pathologies.

Conclusions

The contribution of this work belongs to the field of computational neurosciences and the design of
serious games. Empirically, it was possible to validate the predominant activity related to executive
functioning while using certain mechanics that were implemented in videogames, with a transversal
contribution of this paper being the taxonomy that describes the main mechanics present in videogames after analysing some of the more common elements in commercial games. Since the contribution of this manuscript cannot be considered a final tool for supporting cognitive diagnosis and reha-

bilitation, the taxonomy and the findings related to correlation between mechanics and cognitive
skills, can be useful for proper design of serious games, conceptually.
It has been proven a clearly prominent activity at the brains prefrontal level that is related to executive functions, carried out in three of the five analysed mechanics. The two mechanics that do not fall
under this category are the accurate actionand the mimic sequences mechanics, during which the first
case, the brain activity is more global and with a higher presence of alpha waves, and the second case
has a greater activation of the temporal lobe. Having evidenced the prominent EF-related activity at
brain level, it can be corroborated the first hypothesis that stated the development of EF using action
videogames, particularly videogames including the three mentioned mechanics. Such development is
held up with classical theories that state the development of cognitive skills though the continuous use
of related brain components [60] and more recent experimental studies in that matter, e.g. [61].
In regards to the second hypothesis related to the identification the cognitive skills developed while
using certain videogame mechanics, it was observed that the relationship is not disjointed (i.e., a mechanic develops several cognitive skills, and each skill can be developed by several mechanics). Regardless, the prevalence of particular cognitive skill development and for each game mechanic has
been proved, as seen in Figure 7. The most correlated pairs are Trails-Timely Action, WashersLearning Patterns, and Interference-Logical Puzzles. Each game mechanic works several cognitive
skills, as the mechanics have similarities with several assessments. Some fewer correlations are:
Trails-Logical Puzzles, Washers-Logical Puzzles, and Interference-Learning patterns. When it comes
to the strength of correlation, Logical Puzzles and Learning Patterns are the most strongly linked with
executive function. We also obtained quantitative values that help the design of serious games with
health purposes, in particular for the support of the diagnosis and treatment of cognitive-related pathologies. Based on those results (Table 3) and as a summary of the contribution of this manuscript,
Table 4 shows the learned lessons and findings in terms of design guidelines to develop videogames
related to cognitive development:
Table 4. Design guidelines to develop serious videogames with cognitive treatment and evaluation purposes.
Requirement

Involved mechanics

Description

General cognitive

Accurate

This mechanics activate the brain the global form

activation

logical puzzles

action

and

due to improve some abilities located in different

parts of brain, (memory, inhibition, problem solving, etc.)

Executive Function

Learning patterns and

Mainly, this mechanics improve frontal lobe and

logical puzzles

the abilities related in.

Develop Working

Mimic sequences, logi-

Memory is present in many mechanics (working

Memory

cal puzzles and timely

memory) but in this specific is more necessary

action

use it (short and long term memory).

Develop Shifting

Logical

puzzles

and

This mechanics need more attention and block out

Develop Inhibition

timely action

information not necessary in that moment.

Mimic sequences, ac-

In this mechanics is very important to be control

curate action and pat-

of the right things that you can do and not in the

terns learning

rest.

Acknowledgments. This work was conducted in the context of UBIHEALTH project under International Research Staff Exchange Schema (MC-IRSES 316337). Thanks to the students: Laura
Gutierrez, Rodrigo Marn, Carlos Vallejo, Carlos Villa, and M Jess Ciudad for the development of
those amazing videogames. Finally, and especially, many thanks to all families involved in this project.

References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

, U., & Kerns, K. A. The development of executive function, in Handbook of Child Psy, and R. M.
Lerner (Hoboken, NJ: John Wiley and Sons), 571-623. (2015).
Carlson, S. M., Zelazo, P. D., & Faja, S. Executive function, in Handbook of Developmental
Psychology, ed P. D. Zelazo (New York, NY: Oxford University Press), 313 743. (2013).
Miyake, A., & Friedman, N. P. The Nature and Organization of Individual Differences in Executive Functions: Four General Conclusions. Current Directions in Psychological Science, 21(1), 8
14. http://doi.org/10.1177/0963721411429458. (2012).
Diamond, A. Executive Functions. Annu. Rev. Psychol. 64, 135-168. doi: 10.1146/annurevpsych113011-143750. (2013).
Loe, I. M., Chatav, M., & Alduncin, N. Complementary assessments of executive function in preterm
and
full-term
preschoolers.
Child
Neuropsychol.
21,
331-353.
doi:10.1080/09297049.2014.906568. (2015).
Ng, F. F., Tamis-LeMonda, C., Yoshikawa, H., & Sze, I. N. Inhibitory control in preschool predicts early math skills in first grade: Evidence from an ethnically diverse sample. Int. J. Behav. 39,
139-149. doi: 10.1177/0165025414538558. (2015).
de Vries, M., & Geurts, H. Influence of autism traits and executive functioning on quality of life
in children with an autism spectrum disorder. J. Autism Dev. Disord. 45, 2734-2743. doi:
10.1007/s10803-0152438-1. (2015).
Ezpeleta, L., & Granero, R. Executive functions in preschoolers with adhd, odd, and comorbid
adhdodd: Evidence from ecological and performancebased measures. J. Neuropsychol. 9, 258270. doi: 10.1111/jnp.12049. (2015).
Niloufar, J. M., & Reza, K. N. The role of executive functions in bilingual children with reading
difficulties. Scand. J. Psychol. 56, 297-305. doi: 10.1111/sjop.12198. (2015).
McNamara, J. P. H., Reid, A. D., Balkhi, A. M., Bussing, R., Storch, E. A., Murphy, T. K., et al.
Self-regulation and other executive functions relationship to pediatric OCD severity and treatment
outcome. J. Psychopathol. Behav. Assess. 36, 432-442. doi: 10.1007/s10862-014-9408-3. (2014).
Myers, S. G., & Wells, A. Early trauma, negative affect, and anxious attachment: The role of metacognition. Anxiety Stress Coping. 28, 634-649. doi: 10.1080/10615806.2015.1009832
Sastre-Riba, S. (2006). Early development and learning conditions: the role of the executive functions. Revista de Neurologa, 42(2), 43-51. (2015).
Luciana, M. & Nelson, C.A. The functional emergence of prefrontally-guided working memory
systems in four to eight year-old children. Neuropsychologia, 36(3), 273-293. (1998).
McFarlane, A., Sparrowhawk, A. & Heald, Y.: Report on the educational use of games: An exploration by TEEM of the contribution which games can make to the education process.(2002).
Thompson, T., Baranowski, R., Buday, J., Baranowski, V. Thompson, R. Jago., & Griffith, M.J. :
Serious Video Games for Health: How Behavioral Science Guided the Development of a Serious
Video Game. Simulation & Gaming, vol. 41. pp 587 606. (2010).

16.

17.
18.
19.
20.
21.
22.
23.
24.

25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.

Velazquez, A., Martinez-Garcia, A. I., Favela, J., Hernandez, A., & Ochoa, S. F. Design of
exergames with the collaborative participation of older adults. Paper presented at the Proceedings
of the 2013 IEEE 17th International Conference on Computer Supported Cooperative Work in
Design, CSCWD 2013, 521-526. doi:10.1109/CSCWD.2013.6581016. (2013).
Gorini, A., Gaggioli, A., Vigna, C., & Riva, G. A second life for eHealth: Prospects for the use of
3-D virtual worlds in clinical psychology. Journal of Medical Internet Research, 10(3)
doi:10.2196/jmir.1029. (2008).
Acampora, G., Cook, D. J., Rashidi, P., & Vasilakos, A. V. A survey on ambient intelligence in
healthcare. Proceedings of the IEEE, 101(12), 2470-2494. doi:10.1109/JPROC.2013.2262913.
(2013).
Bavelier, D., Green, C. S., Schrater, P., & Pouget, A. Brain plasticity through the life span: Learning to learn and action video games. Annual Reviews of Neuroscience, 35, pp 391-416. (2012).
Andreassi, J. L. Human behavior and physiological response. Psychophysiology, 4, 1-12. (2000).
Anderson, P.: Assessment and Development of Executive Function (EF) During Childhood. Child
Neuropsychology, vol. 8, pp 71-82. (2002).
Portellano, J. A., Martnez, A., & Zumrraga, A. : Evaluacin neuropsicolgica de las Funciones
Ejecutivas en nios (Neuropsychological assessment of executive functions in children): ENFEN.
TEA Ediciones, Madrid. (2009).
L.A. Rabin, W.B. Barr, & L.A. Burton. Assessment practices of clinical neuropsychologists in the
United States and Canada: A survey of INS, NAN and APA Division 40 members. Archives of
Clinical Neuropsychology, 20, pp. 3365, (2005).
Lezak, M. D. Neuropsychological assessment (3rd ed.). New York: Oxford University
Press.Bausela. E & Santos, J.L. (2006). Utilidad del stroop en la psicologa clnica. Advances in
relational mental health. Vol.5, nm.1. ASMR Revista Internacional On-line - Dep. Leg. BI-282401 - ISSN 1579-3516, (1995).
Spreen, O., & Strauss, E. A compendium of neuropsychological tests: Administration, norms and
commentary (2nd ed.). New York: Oxford University Press. (1998).
Simon, H. A. The functional equivalence of problem solving skills. Cognitive Psychology, 7, 268288. (1975).
Goel, V. & Grafman, J. Are the frontal lobes implicated in "planning" functions? Interpreting data
from the Tower of Hanoi. Neuropsychologica, 33, 623-642. (1995).
Duvinage, M., Castermans, T., Petieau, M, Hoellinger, T., Cheron, G., & Dutoit, T. Performance
of the Emotiv Epoc headset for P300-based applications. BioMedical Engineering OnLine,
(2013).
M. Murugappan, N. Ramachandran, & Y. Salzail. "Classification of human emotion from EEG
using discrete wavelet transform," J. Biomedical Sci. and Eng, vol. 3, no.4, pp. 390-396. (2010).
E. T. Mampusti, J. S. Ng, J. J. I. Quinto, & G. L. Teng. "Measuring academic affective states of
students via brainwave signals," KSE, pp. 226-231. (2001).
Ray, W. J., & Cole, H. W. (EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science, 228, 750-752. doi:10.1126/science.3992243.
(1985).
Aftanas, L. I., & Golocheikine, S. A. Human anterior and frontal midline theta and lower alpha
reflect emotionally positive state and internalized attention: High-resolution EEG investigation of
meditation. Neuroscience Letters, 310, 57-60. doi:10.1016/S0304-3940(01)02094-8. (2001).
Fontecha, J., Navarro, F. J., Hervs, R., & Bravo, J. Elderly frailty detection by using accelerometer-enabled smartphones and clinical information records. Personal and ubiqui tous computing,
17(6), 1073-1083 (2013).
Raso, I., Hervs, R., & Bravo, J. m-Physio: personalized accelerometer-based physical re- habilitation platform. In: Proceedings of the Fourth International Conference on Mobile Ubiquitous
Computing, Systems, Services and Technologies, pp 416-421. Florence, Italy (2010).
Hervs, R., Fontecha, J., Ausn, D., Castanedo, F., Bravo, J., & Lpez-de-Ipia, D. Mobile Monitoring and Reasoning Methods to Prevent Cardiovascular Diseases. Sensors. 13(5), 6524-6541.
MDPI (2013).
Whyte, E. M., Smyth, J. M., & Scherf, K. S.: Designing serious game interventions for individuals with autism. Journal of Autism and Developmental Disorders, (2014).

37.
38.
39.

40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.

Garris, R., Ahlers, R., & Driskell, J. E.: Games, motivation, and learning: A research and practice
model. Simulation Gaming, vol 33, pp 441467 (2002).
Kato P. M., Cole S. W., Bradlyn A. S., & Pollock B. H.: A video game improves behavioral outcomes in adolescents and young adults with cancer: A randomized trial. Pediatrics vol.122, pp
305317 (2008).
Diehl, L. A., Souza, R. M., Alves, J. B., Gordan, P. A., Esteves, R. Z., Jorge, M. L. S. G., & Coelho, I. C. M.: InsuOnline, a Serious Game to Teach Insulin Therapy to Primary Care Physicians:
Design of the Game and a Randomized Controlled Trial for Educational Vali-dation. JMIR Research Protocols, 2(1), e5 (2013).
Synnott, J., Chen, L., Nugent, C. D., & Moore, G. WiiPD-objective home assessment of
parkinsons disease using the nintendo wii remote. IEEE Transactions on Information Technology
in Biomedicine, 16(6), 1304-1312. doi:10.1109/TITB.2012.2215878. (2012).
Brezinka, V. & Hovestadt, L.: Serious games can support psychotherapy of children and adolescents, pp. 357-364. Springer, Heidelberg (2007).
Azrulhizam Shapi'i, Nor Azan, M. Z. & Ahmed, M. E.: A game system for cognitive rehabilitation. BioMed Research International, (2015).
Sourina O, Wang Q, & Nguyen MK.: EEG-based Serious Games and Monitoring Tools for Pain
Management. Stud Health Technol. Inform (2011).
Baek, Y., Ko, R., & Marsh, T.: Trends and Applications of Serious Gaming and Social Media.
Springer, Heidelberg (2014).
Green,C.S. & Bavelier,D. Actionvideogamemodifiesvisual selectiveattention. Nature 423, 534
537 (2003).
Green,C.S., Pouget,A. & Bavelier,D..Ageneralmech- anism forlearningwithaction videogames:improvedprobabilis- ticinference. Curr.Biol. 20, 1573 15792 (2010).
Kendall, P.: Coping Cat Workbook. Temple University, Philadelphia (1990).
Cowan, J.D. & Markham, L.: Eeg biofeedback for the attention problems of autism - a case study.
Biofeedback and Self-Regulation 19(3), 287287 (1994).
Kerson, C., Sherman, R.A. & Kozlowski, G.P.: Alpha suppression and symmetry training for generalized anxiety symptoms. Journal of Neurotherapy 13(3), 146155 (2009).
Carrillo, I., Meza-Kubo, V., Morn, A. L., Galindo, G., & Garca-Canseco, E. (2015). Emotions
identification to measure user experience using brain biometric signals doi:10.1007/978-3-31920892-3_2
Wang, Q., Sourina, O., & Nguyen, M. K. Fractal dimension based neurofeedback in serious
games. The Visual Computer, 27(4), 299-309. doi:10.1007/s00371-011-0551-5. (2011).
M. S. Cameiro, S. Bermdez i Badia, L. Zimmerli, E. Duarte Oller, & P. F. M. J. Verschure, "A
Virtual Reality System for Motor and Cognitive Neurorehabilitation," in Association for the Advancement of Assistive Technology in Europe - AAATE 2007 San Sebastian, Spain, (2007).
Diamond A, & Lee K. Interventions and programs demonstrated to aid executive function development in children 412 years of age. Science. 333:95964. (2011).
Diamond, A.. Executive Functions. Annu. Rev. Psychol. 64, 135-168. doi: 10.1146/annurevpsych113011-143750. (2013).
Wass S, Porayska-Pomsta K, & Johnson MH. Training attentional control in infancy. Curr. Biol.,
21:15. (2011).
Bisoglio, J., Michaels, T. I., Mervis, J. E., & Ashinoff, B. K. Cognitive enhancement through action video game training: great expectations require greater evidence. Frontiers in Psychology, 5,
pp 136. (2014).
Baek, Y., Ko, R. & Marsh, T.: Trends and Applications of Serious Gaming and Social Media.
Springer, Heidelberg (2014).
Puthankattil Subha, P., Joseph, P.K., Acharya U,R. & Lim, CH, M.: EEG Signal Analysis: A
Survey. J Med Syst, vol. 34, pp 195-212. Springer, Heidelberg (2010).
Banos, O., Damas, M., Pomares, H., Prieto, A., & Rojas, I. (2012). Daily living activity recognition based on statistical feature quality group selection. Expert Systems with Applications, 39(9),
8013-8021. doi:10.1016/j.eswa.2012.01.164
Lamarck, J.B. (1809). Philosophie zoologique. 262 pp. Valencia, ca.1910.
Mae-Wan Ho. Genetic engineering: dream or nightmare? Turning the tide on the brave new world
of bad science and big business. (2000) New York, NY: Continuum. ISBN 0-8264-1257-2

HIGHLIGHTS

Correlation between Videogames Mechanics and Executive Functions through

EEG analysis by Mondejar et al.
- Typical videogame mechanics exercise the prefrontal brain area, making it possible to improve cognitive skills through the design of serious games.
- There are particular videogame mechanics with strong correlation to specific cognitive processes, such as inhibition, working memory, and shifting.
- The combination of neurosciences and computation makes possible the improvement of diagnosis and treatment of cognitive issues.

Ramn Hervs (Corresponding author)

Ramon.hlucas@uclm.es