Comm-ICU: a Multisensory Brain Computer Interface for Binary

Communication in the Intensive Care Unit

James McLean, Wyatt Bertorelli, Laurel McCallister, Samuel Rendall, Ginamarie Spiridigliozzi
Faculty Advisor: Deniz Erdogmus
PhD Student Mentor: Fernando Quivira
April 2016

1 Problem statement and Background

Patients who are required to spend time in the intensive care unit due to a life threatening condition are prone
to develop delirium and assessment of their mental state requires communication with the doctors. Many of these
patients have suffered from trauma resulting in extreme loss of voluntary motor control, termed locked-in syndrome [1,
2]. The procedures often used by doctors to assess a patient’s mental state are ineffective on those with locked-in
syndrome since these methods rely on conversation between the doctor and the patient. Caregivers are often available
to help these patients communicate; however, most hospitals do not have the staff to support having a caregiver
present for every test. Additionally, some patients are too severely disabled to communicate effectively, even with
the help of a caretaker [3]. Currently, there is no single method for communication with these patients that is robust
enough to handle the wide variety of trauma experienced by ICU patients. This is problematic because cognitively
aware patients who are unable to communicate are unable to undergo mental health screenings placing them at risk
for developing delirium and other preventable mental illnesses, ultimately worsening their chances for a successful
recovery.

2 Methods and Solutions Considered

By providing a means for physicians to communicate with ICU patients who are unable to speak, or even move, our
system could enable physicians to diagnose delirium in even the most severely disabled patients. Early detection
leads to early intervention, ultimately improving ICU mortality rates as delirium prevalence falls [2]. Intervention can
involve changing the patient’s environment by reducing noise, providing orientation through clocks and calendars, as
well as stopping doses of certain medications and checking for delirium related diseases [2]. Brain computer interfaces
(BCIs) have shown the potential to restore autonomous actions by allowing users to communicate and control devices
through the measurement of electrical signals from their brain activity [4]. BCIs have been tested using many forms
of stimulus, most commonly with the visual modality [5, 6]. Patients in the ICU, however, are likely to have been
a victim of trauma which has impaired their vision capabilities. We propose a multisensory stimulus framework for
binary BCI-based communication interfaces. With visual, tactile, and auditory stimuli, BCI applications can reach
patients with a wide range of debilitating injuries including those with impaired sight and/or hearing.
Specifically, we have developed a multi-sensory (tactile, visual, and auditory) binary communication brain com-
puter interface (BCI) using electroencephalography (EEG) signals. By observing a patient’s EEG during presentation
of visual, tactile, or auditory stimuli that encode a ”yes” or ”no” response paired with questions with unambiguous
answers, the system constructs a model of the prior probabilities of features seen in the patient’s EEG during this
training phase. Using this model, the system can infer a patient’s intent by detecting similar features in their EEG
during subsequent communication sessions. The system consists of the necessary hardware for brain stimulation, as
well as the software needed to perform intent detection. A detailed analysis of the main components of the design is
described below.

3 Comm-ICU Design
To maximize the flexibility of our design, we have followed the human-in-the-loop cyber-physical system design
framework [7]. This framework has three sub-systems: (1) human interaction through physiological signal extraction,
user interface, and stimulation, (2) fusion of physiological and non-physiological evidence for intent inference, and

1
 Figure 1: Human-in-the-loop cyber-physical system diagram applied to the communication interface in the ICU

(3) interaction of the system with the physical environment (fig. 1). Sub-systems and modules transfer information
with each other via a unified communication scheme. The framework’s modular design simplifies both upgrading as
well as building new extensions to the underlying applications.
Patient responses are inferred by detecting Event Related Potentials (ERPs) and Stimulus Evoked Potentials
(SEPs) in a patient’s EEG signals. ERPs and SEPs are modulated by attention, so by focusing on stimuli that
correspond to either “yes” or “no”, patients can control when ERPs and SEPs occur. A machine learning pipeline
is then used to detect ERPs and SEPs and determine what the patient’s response is most probable. This system
is multi-modal, employing visual, auditory, and tactile stimuli to accommodate the diversity of injuries in an ICU
environment.

Figure 2: Multisensory stimulus system design.

3.1 Control Interface

The multisensory stimulus module design is shown in fig. 2. High level commands are sent from the host (BCI
application) to the control interface. A Beaglebone Black is used to control the stimuli and communicate with
the main BCI application. The network interface was implemented using OpenDDS, a real-time publish-subscribe
communication module [8]. Through device virtualization, many “devices” can exist on the same hardware, while
remaining conceptually distinct. DDS decouples modules and empowers anonymous communication thus encouraging
a modular design of applications. The host communicates with the stimulus module by instantiating objects for
predefined topics corresponding to each modality.

3.2 Stimulus Hardware

The hardware component of focuses on presenting on driving visual, audio, and vibro-tactile stimuli to a patient.
Timing constraints require the hardware to send precise start-of-stimulation events (or triggers) to the data acquisition
component, and delivering uncorrupted stimuli to the patient. The low-level implementation for each modality is

2
shown in fig. 3. The visual stimulus is presented with a set of LED arrays (4 channels) driven by custom platform
(cape) with a Xilinx Spartan3E FPGA. The BCI developer can configure run-time frequency, pattern and brightness
with pulse width modulation (PWM) for each stimulus channel. Similarly, the tactile module is driven by the same
FPGA, thus allowing users to configure and send vibration waveforms to the C-3 tactors (4 channels). The digital
hardware on the FPGA for tactile stimuli consisted of a single port block ram node and two counters. The ram node
is used as a buffer for programming the tactor waveform with capability for burst writing from the SPI interface.
The first counter is for controlling the sample rate out of the buffer. The second counter mandates the period of the
PWM which is compared against the sample value of the buffer to determine the duty cycle of the PWM. The FPGA
has an extra output channel that connects directly to the data acquisition module as an external trigger signal.
The audio stimulus is driven directly by the Beaglebone Black. The audio waveforms are predefined and stored in
internal memory. Then, the application only sends configuration parameters and start/stop timings to the control
interface. In addition, this module embeds an inaudible high frequency tone inside the stimulus audio waveform to
act as a trigger. A custom PCB was built with a fourth order Butterworth bandpass filter, half-wave rectifier, and
DC voltage amplifier to detect the high frequency tone and send a trigger to the data acquisition system.

3.3 Intent Inference

EEG was sampled 512 Hz using a g.USBamp biosignal amplifier and active g.Butterfly electrodes (G.tec, Austria).
The electrodes were placed on a g.GAMMAcap at positions Fz, FC1, FC2, Cz, C3, C4, CP1, CP2 for tactile and
at Oz for visual paradigm, as determined by the International 10/20 System. The EEG signals were streamed to
MATLAB (The Mathworks, Inc.) and digitally bandpass filtered (FIR, linear phase, passband [0.5, 45]Hz).
For the tactile application, the corresponding EEG response was taken from stimulus onset to 500 ms after,
considering the group delay of the FIR filter appropriately. Principal component analysis (PCA) was used on each
individual channel to discard uninformative signal components (those with extremely small energy). The resulting
vectors were concatenated to build the EEG feature vector yt for corresponding trial t. After pre-processing, the EEG
feature vector yt is mapped from Rn to R through quadratic discriminant analysis (QDA), a quadratic projection
which arises from the log likelihood ratio of two Gaussian class distributions [5]:

G(yt ; µ1 , Σ1 ) π1
xt = ln (1)
G(yt ; µ0 , Σ0 ) π0

where G(·, µi , Σi ) is the multivariate normal density function with mean µi , covariance Σi , and prior πi [9]. It
is common in brain computer interfaces to have high data dimensionality and small training sets. Therefore, the
maximum likelihood estimate for the individual class covariance matrices will be singular or close to singular. One
alternative to solve this problem is to use regularization and shrinkage to obtain better-conditioned covariance
matrices for the class density models:
γ
Σ̂i (λ, γ) = (1 − γ)Σ̂i (λ) + tr{Σ̂i } (2)
n
(1 − λ)Ni Σ̂i + λN Σ̂
Σ̂i (λ) = (3)
(1 − λ)Ni + λN
P
where Ni is the number of trials corresponding to class i, N = i Ni , and Σ̂i is the sample class covariance estimate.
For the visual application, we used CCA to extract the relevant EEG features. The standard CCA approach,
introduced by Lin et al. [10], consists of computing the maximum Pearson’s correlation between a linear combination
of all available channels of EEG data, E ∈ Rd×T fs , and a linear combination of template signals (Yi ) [11]:

ρi = max ρ(αE, βYi ) (4)

α,β

where ρ(·, ·) refer to Pearson’s correlation function between two vectors of samples, fs is the sampling frequency, T
is the trial length in seconds, and d is the number of EEG electrodes used. The template signal Yi is derived from
the assumption that EEG can be approximated by a few sinusoids under SSVEP; therefore, for each fi ∈ F , the
template Yi is built as:  
sin(2πfi t)
 cos(2πfi t) 
 
..
Yi =   ∈ R2H×T fs (5)
 
 . 
 sin(2πHfi t) 
cos(2πHfi t)

3
for t = [ f1s , f2s , . . . , T ], and where H is the number of harmonics considered. We can then define our CCA feature
vector as the set of individual CCA cofficients for each stimulation class:

xt = [ρ1 , . . . , ρM ]T (6)

In both scenarios, inference is performed with probabilistic models based on kernel density estimation. The
conditional probability density functions of EEG features given the class label are given by [12, 13, 14]:
1 X
fˆEEG (x|c = c) = G(x; xk , σc2 ) (7)
|Nc |
k∈Nc

where Nc corresponds to the set of indices for the EEG trials acquired during training. Here, xk is the EEG feature
for the k-th trial from the training set, σc is the kernel bandwidth parameter that determines how smooth the KDE
will be (bias-variance trade-off) – obtained using Silverman’s rule [13].

Figure 3: Module implementation of multisensory stimulus framework.

3.4 User Interface

Since our device is designed to be used by non-technically oriented physicians and caregivers, an interface was designed
to provide easy control of the system. Currently, the GUI consists of 2 primary screens. The first, shown in Figure
4, simply displays the three BCI modalities available and the user clicks to select the desired mode. Afterwards,
the interface transitions to the main display, shown in Figure 5. This part of the interface is where the caregiver
can begin the test. By pressing the appropriate buttons, the system can either begin the calibration session, run
through the actual CAM ICU test, or cancel the session at any time. Additionally, the questions being presented
are displayed for the user alongside the true answers, the patient’s response, and the patient’s overall assessment. In
future versions of the GUI, we plan to include a developer mode along side this interface which will allow the software
to be used by engineers and easily updated with new classification paradigms as research progresses.

Figure 4: Window for choosing which of the three modalities the user would like to use.

4
Figure 5: Shows the progress of the assessment by displaying questions, answers, and the patient’s response. Includes
buttons for running calibration and testing sessions.

Outcome
Testing classification performance on multiple neurotypical subjects was performed in order to validate the function-
ality of the system as a multi-modal BCI. Data was collected and classified in both the visual and tactile modalities.
The visual paradigm required the subject to stare, one at a time, at two arrays of LEDs flashing, one flashing
at 8.1 Hz and the other at 10.5 Hz. The training data was collected for one minute, 30 seconds for each frequency,
before classification. As expected, the system performed extremely well, correctly classifying 10 binary questions
with close to %99 confidence for all subjects. Each question was classified using three seconds of EEG data. The
power spectral density of the subject’s EEG signal for one response is shown in Figure 6.

Figure 6: PSD of EEG signal at channel Oz. The dashed red lines mark the region around the target frequency of
8.1 Hz, where a peak is visible

Tactile stimulation was delivered using two vibrotactile devices, held in each of the subject’s hands. Each stimulus
was defined by a short burst of vibration at 200 Hz for 0.35 seconds, followed by an inter-stimulus pause of 0.15
second duration. The order in which the right and left tactors are stimulated is generated pseudo-randomly; target
and non-target stimuli are equally probable and these stimuli occur one in every four trials on average (with the rest
of the stimuli acting as distractors).
For the tactile test, each subject received 600 stimuli, 300 for each class, totalling at a 300 second training session.
Each subject was asked 10 questions, and the system attempted to classify each question from 50 total stimulations.
The resulting ERP averaged over all trials for one of these questions is shown in Figure 7. The system was able to
classify 6 out of the 10 questions correctly for all subjects except for one, who answered only half correctly.
The tactile paradigm, being ERP based, provides a much more difficult classification challenge. This is primarily
due to the fact that ERP’s are a higher order cognitive function and are therefore much more difficult to illicit
reliably. Tactile based BCI offers the additional challenge of being very difficult for a subject to remain attentive to
the target stimulus and ignore the non-target stimulus when compared to visual or even auditory stimulus. These are
most likely the reasons for the system’s classification performance, however, we are continuing to work on developing
a robust paradigm suited to the challenges of tactile ERP based classification.

5
Figure 7: Average ERP over all trials from channel Cz. The red line indicates onset of the P300 response at 300 ms.

Cost
Below is a table outlining the cost for our device. Some of these options, most notably the laptop computer and the
EEG amplifier and related equipment have cheaper options which would not affect the functionality of the device
as a whole. Additionally, the wheeled lectern stand is provided not only so that caregivers can easily maneuver the
device throughout the hospital room, but also so that a single device can be used to service many patients. All parts
of the device which the patient potentially comes in contact with (e.g. headphones, tactors, electrodes, EEG cap)
are all either disposable or easily cleaned so that sanitation is not an issue for multi-patient use.

Item Cost of research prototype ($) Cost of production prototype ($)

Laptop Computer 1000 1000
BeagleBone Rev C 49.99 49.99
Custom PCB 841.95 841.95
Custom Enclosure 158.79 158.79
Auxiliary Parts 285 285
EEG Equipment 12000 1000
Total 14335.73 3335.73

Significance
We have built a BCI system that interfaces with the three main senses of the human body to accommodate a patient
with any kind of sensory impairment and tailor the communication interface to his or her needs. This system could
potentially enable these patients to participate in delirium tests in the ICU. The system is designed modularly
with future enhancements and alternative applications in mind, so it can be extended further. We successfully
demonstrated the capabilities of our prototype in both visual and tactile BCI paradigms. The system can potentially
be used with other vibro-tactile paradigms. In future work, we plan to use this stimulus framework to investigate
the optimal tactile and auditory paradigms that can maximize communication rates with users.
This device has the added feature of being designed specifically for compatibility with the ICU environment.
Through consultation with a physician at the Oregon Science and Health University, we have incorporated mechanical
features, a GUI designed to be easy to use for both technically and non-technically oriented users, and a module
for facilitating the CAM ICU test which acts as the standard procedure for delirium assessment in the ICU nation-
wide. Our mechanical design specifically allows the device to be easily sanitized, transported, and adjustable for
comfortable use by both the caretaker and patient. Additionally, the EEG equipment has been tested in the ICU
environment with neurotypical users and verified to be unaffected by potential interference from other commonly
found electrical equipment. To our knowledge, this device is the first BCI system to be designed with this level of
comprehensive ICU compatibility.

Acknowledgements
Our work is supported by NSF (CNS-1136027, IIS-1149570, CNS-1544895), NIDRR (90RE5017), and NIH (R01DC009834).

6
References
[1] Jerome Posner, Clifford Saper, Nicholas Schiff, and Fred Plum. Plum and Posner’s Diagnosis of Stupor and
Coma. Oxford University Press, 4th edition, 2007.
[2] E. W. Ely. Delirium as a Predictor of Mortality in Mechanically Ventilated Patients in the Intensive Care Unit.
JAMA: The Journal of the American Medical Association, 291(14):1753–1762, 2004.
[3] John M Costello, Lance Patak, and Jennifer Pritchard. Communication vulnerable patients in the pediatric
ICU: Enhancing care through augmentative and alternative communication. Journal of pediatric rehabilitation
medicine, 3(4):289–301, jan 2010.
[4] L a Farwell and E Donchin. Talking off the top of your head: toward a mental prosthesis utilizing event-related
brain potentials. Electroencephalography and clinical neurophysiology, 70:510–523, 1988.
[5] Umut Orhan. RSVP Keyboard: An EEG Based BCI Typing System with Context Information Fusion. PhD
thesis, Northeastern University, 2013.
[6] Hubert Cecotti. A self-paced and calibration-less SSVEP-based brain-computer interface speller. IEEE trans-
actions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine
and Biology Society, 18(2):127–33, apr 2010.
[7] Gunar Schirner, Deniz Erdogmus, Kaushik Chowdhury, and Taskin Padir. The Future of Human-in-the-loop
Cyber Physical Systems. IEEE Computer, 46(1):1–8, 2013.
[8] Object Computing, Inc. OpenDDS — Object Computing, Inc. 2013.
[9] AJ Isenmann. Modern multivariate statistical techniques. Springer, 2008.
[10] Z. Lin, C. Zhang, W. Wu, and X. Gao. Frequency recognition based on canonical correlation analysis for
ssvep-based bcis. IEEE Transactions on Biomedical Engineering, 54(6):1172–1176, June 2007.
[11] Guangyu Bin, Zhonglin Lin, Xiaorong Gao, Bo Hong, and Shangkai Gao. The ssvep topographic scalp maps
by canonical correlation analysis. In 2008 30th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society, pages 3759–3762, Aug 2008.

[12] Trevor. Hastie, Robert. Tibshirani, and JJH Friedman. The elements of statistical learning. Springer, 2011.
[13] BW Silverman. Density estimation for statistics and data analysis. Springer, 1986.
[14] CM Christopher Bishop and NM Nasrabadi. Pattern recognition and machine learning. Springer, 2006.