How about taking a Low-cost Multi­modal Bio-sensing and
Eye-gaze Tracking System into the “Wild”?
Siddharth, Tzyy-Ping Jung, and Terrence J. Sejnowski
Abstract— We present a low-cost wearable multi-modal
bio-sensing system capable of recording bio-markers and
eye-gaze overlaid on world view in real-world settings. The
system incorporates a wearable computer (e.g. Raspberry Pi 3)
to collect data from bio-sensors (e.g. 1-channel ECG and
multi-channel EEG), and two cameras (a world camera and an
IR-based eye camera). The wearable computer transmits data
from all the sensors to a main computer over Wi-Fi, while
time-stamping each packet data stream. The system is easy to
wear, comfortable, and scalable to incorporate bio-sensors as
per the need.
I. INTRODUCTION
Presently, bio-sensors are confined to laboratory’s
restricted environments for experimental data collection.
There is no robust yet comfortable system that could be used
for data collection in mobile settings or has applications in
real-world environments. Traditional bio-sensing systems are
costly, bulky and not designed for comfort or ease-of-use,
making them impractical for real-world studies [1].
Additionally, the bio-sensors have to be usually assembled
together, which requires more effort in time synchronization
and calibration between them. Another issue is not being able
to exactly pinpoint the events happening in the real world to
which the subject is responding. The usual practice to achieve
this as a roundabout is by asking the subject to mark the event
(for example, by pressing a button). But, this induces further
unreliability and compromises the experiment’s veracity.
Using more than one type of bio-sensor allows for being able
to boost performance by compensating for the other(s). For
example, the bio-markers in neurophysiological data (e.g.
electroencephalogram or EEG) can complement and
supplement those available in physiological data (e.g. ECG,
HR and HRV) for assessing human emotion [2].
II. RESEARCH
We present a low-cost, wireless, multi-modal bio-sensing
system (Figure 1a) which is comfortable to wear, and can
reliably monitor bio-signals such as EEG, ECG, EMG, GSR
etc. in real-time. Furthermore, it also overlays subject’s
eye-gaze over his/her world view so that events in the real
world can be easily tagged with the corresponding bio-signals.
The system is easy to wear (takes less than 5 minutes to set up
*This work was supported in part by the Army Research Laboratory
under Cooperative Agreement Number W911NF-10-2-0022.
Siddharth is with Department of Electrical and Computer Engineering at
University of California San Diego, La Jolla, CA, 92093, USA (phone:
1-650-772-8624; fax: 1-858-822-7556 and email: sid.sapien@gmail.com).
Tzyy-Ping Jung is with Institute for Neural Computation, University of
California San Diego, La Jolla, CA, 92122 (email: jung@sccn.ucsd.edu).
Terrence J. Sejnowski is with Computational Neurobiology Laboratory,
The Salk Institute, La Jolla, CA, 92037 (email: terry@salk.edu).
by oneself), scalable to incorporate bio-sensors as per the
need, can be worn comfortably for long periods of time, and
uses a wireless connection to make it truly mobile. It uses a
wearable computer (e.g. Raspberry Pi 3) to collect data from
bio-sensors (such as 1-channel ECG and 14-channel EEG by
Emotiv), and two cameras (a world camera and an IR-based
eye camera). It transmits the sensors’ data to a main computer
over Wi-Fi, while timestamping each packet data stream. The
main computer is used to visualize the data in real-time while
synchronizing and recording them. After a fast and easy
calibration, the eye camera’s IR image is used to find the
center of the pupil and map it onto the world camera’s view as
the gaze position. The system is scalable as it can be used with
any desired configuration of bio-sensors or sampling rate as
per the target application of the study. Our research currently
focuses on the application of this multi-modal system for
emotion recognition in real-life settings. Real-time processing
of EEG is being done using ICA to extract true brain activity
and measure valence and arousal [3]. These bio-markers
derived from the EEG can be used directly in an augmented
reality (AR) based headset for augmenting visual experience.
For instance, events such as when a person perceives another
person’s face and subsequent emotional response can be
automatically tagged using eye-gaze. The emotional response
to real-world situations can also be visually displayed to the
subject using AR.
a. b.
c.
d.
Figure 1. (a). Multi-modal Bio-Sensing System. (b). ECG Sensor Module.
(c). World Camera. (d). IR-based Eye Camera.
REFERENCES
[1] McDowell K, Lin C-T., Oie K.S., Jung T-P., Gordon S., Whitaker
K.W., Li S-Y., Lu S-W. and Hairston W.D., “Real-World
Neuroimaging Technologies,” IEEE Access 1:131-49, 2013.
[2] Olga Pollatos, Wladimir Kirsch and Rainer Schandry, On the
relationship between interoceptive awareness, emotional experience,
and brain processes, Cognitive Brain Research 25 (2005) 948-962
[3] Yuan-Pin Lin, Jeng-Ren Duann, Jyh-Horng Chen and Tzyy-Ping Jung,
EEG dynamics of musical emotion perception revealed by independent
spectral components, NeuroReport 2010, 21:410–415