DECODING BRAIN COGNITIVE ACTIVITY ACROSS SUBJECTS USING MULTIMODAL
M/EEG NEUROIMAGING
Sarwat Fatima, Awais M. Kamboh, Member IEEE
Neuroinformatics Lab, School of Electrical Engineering & Computer Science (SEECS),
National University of Sciences & Technology (NUST), Islamabad, Pakistan
ABSTRACT
Brain decoding is essential in understanding where and how
information is encoded inside the brain. Existing literature
has shown that a good classification accuracy is achievable in
decoding for single subjects, but multi-subject classification
has proven difficult due to the inter-subject variability. In
this paper, multi-modal neuroimaging was used to improve
two-class multi-subject classification accuracy in a cognitive
task of differentiating between a face and a scrambled face.
In this transfer learning problem, a feature space based on
special-form covariance matrices manipulated with rieman-
nian geometry are used. A supervised two-layer hierarchical
model was trained iteratively for estimating classification
accuracies. Results are reported on a publically available
multi-subject, multi-modal human neuroimaging dataset
from MRC Cognition and Brain Sciences Unit, University
of Cambridge. The dataset contains simultaneous recordings
of electroencephalography (EEG) and magnetoencephalogra-
phy (MEG). Our model attained, using leave-one-subject-out
cross-validation, a classification accuracy of 70.82% for sin-
gle modal EEG, 81.55% for single modal MEG and 84.98%
for multi-modal M/EEG.
Index Terms— Multi-Modal Neuroimaging, Brain De-
coding, Cognitive Activity, M/EEG
1. INTRODUCTION
Brain decoding refers to the ability of identifying neural
patterns generated in response to a given stimulus, and con-
versely, be able to predict the stimulus by observing neural
patterns. There have been several successes in decoding brain
activity for individual subjects, however, one of the biggest
challenges in correctly predicting the stimulus is inter-subject
variability.
Brain decoding across subjects is a difficult task because
of the inter-subject variability, existing due to the structural
and functional differences of their brains. These differences
originate from the capacity of neurons to rearrange them-
selves functionally and structurally owing to the powerful
ability of neural plasticity.
978-1-5090-2809-2/17/$31.00 ©2017 IEEE 3224
Due to this variability, the signals are very specific for
each subject and the cortical dipoles for any brain activity
can be different in different subjects. This variability can
change the underlying distribution of the features from sub-
ject to subject and thus, make it difficult for decoding brain
activity across subjects.
Inter-subject brain decoding can be categorized as a trans-
fer learning problem, i.e. it is needed to transfer knowledge
learned from other subjects, or other experiments of same
subject, to new unseen subjects in order to decode brain ac-
tivity of new subjects. This way the brain activity relevant
to a particular stimulus can be confirmed. Brain decoding
is useful in several applications, such as Brain Computer In-
terface (BCI) applications where an individual uses his brain
activity to interact with an external machine, or for a reha-
bilitation training, such as neurofeedback. In all these appli-
cations, typically, the classifier is calibrated or trained for a
particular subject. However, what is required is a generalized
classifier, trained on brain patterns of other subjects, or other
experiments of the same subject, which could accurately clas-
sify brain activity.
Multi-modal M/EEG can capture information about brain
patterns which may not be available when using EEG or MEG
as a single modality. The underlying neuronal sources for
both MEG and EEG are similar, but both modalities are sensi-
tive to different components of the dipolar sources. The EEG
field can pick up both tangential and radial components while
MEG is only sensitive to the tangential component of the
dipolar sources. Moreover, both MEG and EEG have good
temporal resolution but MEG has slightly better spatial res-
olution than EEG. This complementary information in MEG
and EEG signals is beneficial for better dipole localization
and classification performance.
In this work, we propose a new combination of feature ex-
traction and classification model using multi-modal M/EEG
which caters for inter-subject variability and improves the
classification performance across subjects. To the best of
our knowledge, the combination of special form covariance
matrices as features and supervised hierarchical iterative
classification has not been applied on multi-modal M/EEG.
The proposed model estimates the Leave-One-Subject-Out
(LOSO) two-class classification accuracy for single modal
EEG, single modal MEG and multi-modal M/EEG. A signif- After spatially filtering the input signal, a special form of
icant improvement was observed in the LOSO classification Symmetric Positive Definite (SPD) covariance matrix is esti-
accuracies as compared to the state-of-the-art classification mated to capture the relevant information of the trials. First,
accuracy of 77.4% on the same dataset for single modal MEG a new trial Zi is built by concatenating spatially filtered av-
[1]. erage evoked potential P (k) and the spatially filtered trial zi
[1]. Then, the SPD covariance matrix of the new trial Zi is
estimated as
2. METHODOLOGY
1
This section explains the specific methods used in the fea- Ci = Zi ZiT (2)
N
ture extraction and classification processes for our proposed
The riemannian geometry tools i.e. riemannian mean &
model as shown in Fig. 1.
riemannian distance [1] are used to keep the special structure
of the SPD covariance i.e. matrix form. However, for vector-
2.1. Transfer Learning based classification purposes, the covariance matrice has to be
converted into a euclidean vector. Tangent space mapping is
Transfer learning is a method to improve a classifier in one used for approximation of riemannian distance computations
domain by transferring information from a related domain. into euclidean distance computation using geometric mean of
This method is often useful when the training data is limited, the covariance matrices as reference [1].
expensive to collect or not easily accessible. Brain decoding
across subjects can be categorized as a transfer learning (TL)
Feature Extraction
problem. Transfer learning is formally defined as improving
SPD Riemannian Tangent
the target predictive function given a source domain DS with Spatial Riemannian
Covariance Distance Space
filtering Mean
a corresponding source task TS and a target domain DT with Matrix
a corresponding target task TT . A domain is defined by two
Classification Model
parts, a feature space X and the marginal probability distribu-
Hierarchal model Iterative Training
tion P(X) [2]. SVM Xtest
In our problem, the source and target domain are same i.e. SVM Random Confidence = yProb - 0.5
yPred
human brain but the marginal distribution is different because Forest W = 10 * (confidence > 0.1)
of structural and functional differences across subjects. We SVM yPred
have different feature space for each subject and we want to
improve the classification accuracy on a new subject when Level 0 Level 1

the classifier is trained on other subjects. This was termed as

transductive transfer learning (TTL) in [3]. Fig. 1: Proposed system diagram showing a special-form co-
variance matrix used for features in a two-layer hierarchical
iterative classifier.
2.2. Feature extraction

xDAWN spatial filtering was used for dimensionality reduc-
tion and for maximizing the signal to noise ratio of the evoked
potentials. xDAWN assumes that the signal is composed of
two typical patterns; one evoked by individual class and one
evoked by whole signal i.e. both classes. Therefore, the spa-
tial filters are estimated for class k as
T
wT P (k) P (k) w
w∗ = argmax (1)
wT XX T w
where P is the average evoked potential, w is the spatial
filter vector and X is the matrix of the whole signal. The
spatial filters are selected corresponding to highest eigen
values for each class through eigen value decomposition
(k) (k)T
of P XX P
T . The resulting set of spatial filters from each
class are concatenated in a single matrix W which performs
the spatial filtering operation on both the trials and average
evoked potential [1].
2.3. Classification
The classification model is a two-layer hierarchical combina-
tion of non-linear Support Vector Machine (SVM) and ran-
dom forest (RF) at level 0, and iterative training at level 1 [4].
An SVM can model complex non-linear relationships by
finding a hyperplane between a set of samples belonging to
class labels. A non-linear Radial Basis Function (rbf) ker-
nel was used for separating the linearly non-separable data
points to higher dimensions and creating more accurate deci-
sion boundary. The parameter C = 1 and γ = 1/nf eatures
are used for optimization of rbf kernel where nfeatures is the
total number of features used. The parameter C controls the
misclassification of training examples versus the simplicity of
the decision surface i.e. a higher C results in classifying all
training samples accurately. The parameter γ defines the in-
fluence of training sample by giving a weight to each training
sample. Although SVM uses significant amount of memory

3225
and processing power, using the right kernel can help improve
the speed of the algorithm.
Random Forest classier is an ensemble learning algorithm
which sequentially combines output of several decision tree
classifiers and averages the probabilistic predictions to im-
prove the generalizability or robustness over a single estima-
tor.
2.3.1. Level 0: Hierarchical Model
The two-layer hierarchical combination of SVM classifiers
and random forest classifier takes a 3-D feature space as in-
put. The first dimension is the number of trials while the sec-
ond and third dimension depends on the chosen number of
features. In our case, each subject has a feature subspace
which is aggregated across subjects to form a new feature
space of dimension [trials x single-subject features x across-
subject features]. The 3-D form of the feature space was
kept instead of combining second and third dimension. The
first layer trains a collection of non-linear SVM classifiers
using data from second (single-subject) and third dimension
(across-subject). The total number of SVM classifiers de-
pends on the length of second and third dimension [4]. The
resulting probability estimates from SVM are then fed into a
1000-tree random forest classifier which makes the final pre-
diction on the test subject as shown in Fig. 1.
2.3.2. Level 1: Iterative Training
The two-layer hierarchical model is wrapped inside the iter-
ative learning classifier to transfer knowledge to the test sub-
ject. The predicted labels and corresponding probabilities at
level 0 are used at level 1 to improve the classification perfor-
mance. The LOSO is used to divide the subjects into training
and test subjects. The predicted labels for the test subject at
level 0 are used as ground truth for the test subject at level
1. The predicted probabilities are used for screening the test
subject samples i.e increasing the weight of the most reliable
samples. It is expected that the samples with predicted prob-
ability far from chance probability of 0.5 in case of binary
class are more reliable and therefore, duplicated by the fac-
tor of 10 which was set an optimal value for replication in
[4]. As shown in Fig. 1, the confidence for a given sam-
ple is calculated by subtracting the predicted probability of
that sample from chance probability of 0.5. If the confidence
value is greater than 0.1, the sample is considered reliable and
duplicated by the factor of 10 and if it is less than 0.1, then
the sample is not duplicated. The test subject is retrained with
predicted labels from level 0 and duplicated reliable samples
at level 1 using training model at level 0. The final predictions
are made on the test subject after re-training.
3226
Table 1: Mean LOSO classification accuracies across sub-
jects
Linear Non- Normal. Subject
SVM Linear (Non- 02 Re-
SVM Linear moval
SVM)
EEG 70.12% 70.03% 69.70% 70.82%
MEG 53.74% 80.84% 78.23% 81.55%
Separate M/EEG 61.18% 78.86% 78.91% 80.97%
Joint M/EEG 52.97% 84.63% 79.92% 84.98%
3. RESULTS AND DISCUSSION
This section describes the implementation of the above men-
tioned proposed system, and tests its performance on a public
dataset.
3.1. Dataset description
The proposed model in section 2 is tested on a multi-subject,
multi-modal human neuroimaging dataset for evoked re-
sponses of face stimuli by D. G. Wakeman and R. N. Henson
from MRC Cognition & Brain Sciences Unit, University
of Cambridge [5]. It contains simultaneous recordings of
70-channel electroencephalography (EEG) and 306-channel
magnetoencephalography (MEG) (102 magnetometers and
204 planar gradiometers) from 19 subjects. However, we
used 16 subjects to keep our analysis comparable with [1].
The subjects undertook multiple runs of a large number
of face stimuli i.e. unfamiliar and scrambled faces. The du-
ration of a single trial was between 1.2 and 1.6 seconds with
approximately 295 trials per category. A complete descrip-
tion of the dataset can be found in [5]. After performing the
usual pre-processing steps in Statistical Parameteric Mapping
(SPM) MATLAB toolbox, we extracted epochs for unfamil-
iar face (0) and scrambled face (1) which varied between 289
and 297 epochs. In order to balance the epochs, the smallest
number of epochs available for both stimuli i.e 289 epochs
were kept only.
A 6th-order butterworth band-pass filter was applied be-
tween 1 and 20 Hz. The first 0.5 seconds of the signal before
stimulus was discarded and 0.5 seconds after stimulus was
kept for further analysis. A set of 8 spatial filters were se-
lected i.e. 4 spatial filters for each category. For each subject,
a feature subspace of dimension [136 features x 578 trials] is
aggregated to form a new feature space of dimension [9248
trials x 136 features x 16 subjects]. The new feature space
was used as input at level 0 of classification model forming
a collection of 152 SVM classifiers each using data from a
single subject (136 features) or across subject (16 features).
Table 1 shows the mean LOSO classification accuracies for
across subjects under different situations.
3.2. Linear vs non-linear SVM
The LOSO classification accuracy for linear and non-linear
SVM are compared for EEG, MEG & M/EEG. As shown in
Table 1, non-linear SVM had a significant improvement in
classification accuracy for MEG & M/EEG as compared to
linear SVM. However, linear & non-linear SVM performed
equally well for EEG and had no significant improvement.
This indicates that MEG and M/EEG samples were not lin-
early separable which is why a linear classifier performed
poorly for these modalities. The box plots in Fig. 2 clearly
shows the improvement in MEG as compared to EEG for non-
linear SVM.
3.3. Multimodal M/EEG feature extraction
EEG and MEG were combined in two different ways for
estimating the LOSO classification accuracy. First, EEG and
MEG were concatenated after separately extracting features
i.e. separate M/EEG; Second, EEG and MEG signals were
combined before extracting features i.e. joint M/EEG as
shown in Table 1 and Fig. 2. The LOSO mean classification
accuracy was significantly improved in joint M/EEG using a
non-linear SVM from 78.86% to 84.63%.
1 single modal MEG to 81.55% and for multi-modal M/EEG
Classification Accuracy
0.9
0.8
0.7
0.6
0.5
0.4
EEG MEG Separate M/EEG
Fig. 2: The box plots are plotted for EEG, MEG, separate
& joint M/EEG for non-linear SVM. A box plot is typically
divided into four quartiles representing distribution of the
dataset which, in our case, is classification accuracy across 16
subjects. The four horizontal lines on a single box plot marks
four quartiles. Each quartile represents 25 % of the data dis-
tribution. The upper horizontal line indicates the maximum
classification accuracy while the lowest horizontal line indi-
cates the minimum classification accuracy. The dots outside
of this distribution are outliers. The red line indicates the me-
dian or the middle of the dataset.
The range of signal values for MEG and EEG are different
with MEG measured in femtoTesla (fT) and EEG measured
in microvolts (µV). Therefore, the euclidean norm of the sig-
nal was taken in order to remove any bias in the result. As
described earlier, riemannian mean and distance are not af-
fected by normalization. However, SVM affects the LOSO
classification accuracies for normalized data because SVM is
a distance based classifier and the distances are changed when
signal values are changed. As shown in Table I, the nor-
malization did not have any significant effect on the LOSO
mean classification accuracy for EEG and separate M/EEG
features. However, the mean classification accuracy of MEG
was reduced by 2% and that of joint M/EEG was reduced by
4%. However, the classification accuracy for joint M/EEG
was still higher than separate M/EEG features.
During the analysis of averaged trials of each subject, no
activity was observed for the averaged event related poten-
tials in subject 2 because of too many noisy trials. Since all
other subjects had evoked potentials, subject 2 was removed,
features were re-extracted and classification accuracy was re-
estimated. This improved the classification accuracies. Table
1 shows improved classification accuracies for EEG, MEG,
M/EEG (separate & joint).
4. CONCLUSION
Brain decoding across subjects is a challenging task since
inter-subject variability can reduce the classification accuracy
significantly. The proposed model captured the inter-subject
variability very well because of its good generalization ca-
pabilities across subjects. Compared to the state of the art,
the model improved the LOSO classification accuracy for
to 84.98% but no significant improvement was observed for
EEG. To the best of our knowledge, the method has not been
applied on either separate or joint M/EEG. Using this as a
preliminary result, further exploration can be done for multi-
class brain decoding across subjects. The work can also be
extended with the addition of fMRI signal to M/EEG.
5. REFERENCES
Joint M/EEG
[1] A. Barachant (2014), “Meg decod-
ing across subjects [online],” Available:
https://github.com/alexandrebarachant/DecMeg2014.
[2] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang,
“A survey of transfer learning,” Journal of Big Data, vol.
3, no. 1, pp. 1–40, 2016.
[3] Emanuele Olivetti, Seved Mostafa Kia, and Paolo
Avesani, “Meg decoding across subjects,” in Pattern
Recognition in Neuroimaging, 2014 International Work-
shop on. IEEE, 2014, pp. 1–4.
[4] H. Huttunen (2014), “Hierarchical combination of lo-
gistic regression and random forest [online],” Available:
https://github.com/mahehu/decmeg.
[5] D. G. Wakeman and R. N. Henson, “A multi-subject,
multi-modal human neuroimaging dataset,” Sci. Data
2:150001 doi: 10.1038/sdata.2015.1, vol. 2, 2015.
3227