Motivation What is BlueEyes (and what is not) ?

? Physiological foundations System design System implementation Future improvements Project work summary System demonstration

Human error - a frequent reason of catastrophes and ecological disasters temporal indisposition

weariness oversight Monitoring of conscious brain involvement Capable sensors available

Motivation What is BlueEyes ?

Physiological foundations
System design System implementation

Future improvements
Project work summary System demonstration

The complex solution for human-operator monitoring: conscious brain monitoring triggering user-defined actions data recording

What is BlueEyes not?

Doesnt predict nor interfere with operators thoughts Cannot force directly the operator to work

Motivation What is BlueEyes ?

Physiological foundations
System design System implementation

Future improvements
Project work summary System demonstration

Generic control rooms Power station Flight control centers Operating theatres anesthesiologists

Common application A simpler system version for drivers

Motivation What is BlueEyes ?

Data Acquisition Unit

Central System Unit

Physiological foundations
System design Objectives System overview DAU CSU System implementation Future improvements Project work summary System demonstration

Atmel 89C52 Bluetooth microcontroller Jazz Multisensor

Bluetooth

Connection Visualization Module Manager Data Analysis Data Logger

Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system.

Motivation What is BlueEyes ?

Data Acquisition Unit

Physiological foundations
System design Objectives System overview DAU CSU System implementation Future improvements Project work summary System demonstration

Atmel 89C52 Bluetooth microcontroller Jazz Multisensor

Lightweight Runs on batteries - low power consumption Easy to use - does not disturb the operator working ID cards for operator authorization Voice transmission using hardware PCM codec

Motivation What is BlueEyes ?

Retina - central vision (macula)

Physiological foundations
System design System implementation

Central vision and the conscious brain

Future improvements
Project work summary System demonstration

Saccade - a rapid eye jump to a new location assigned by the conscious attention process.

Motivation What is BlueEyes ?

Physiological foundations
System design

System implementation

Future improvements
Project work summary

System demonstration

Eye position measuring - direct infrared oculography Oxy- and deoxyhemoglobin measurment Two axial accelerometer Ambient light sensor

Jazz Multisensor

microphone earphone

MC 145483 PCM codec Bluetooth module

Jazz Multisensor

LCD display
LED indicators

UART

MAX232

BlueEyes
ID card interface
ID card

Simple keyboard

Data Acquisition Unit

UART

Beeper

Atmel 89C52 microcontroller

Motivation What is BlueEyes ?

Physiological foundations
System design System DAU CSU Future improvements Project work summary
implementation

System demonstration

Made from scratch by the team Bluetooth module inside Jazz sensor interface Microphone and earphone sockets 5-key keyboard Two lines, 8 characters LCD Powered from 6 AA batteries

Motivation What is BlueEyes ?

Physiological foundations
System design System DAU CSU Future improvements Project work summary
implementation

PCM codec board Microcontroller board

Batteries and Bluetooth module voltage level monitor

System demonstration

Motivation What is BlueEyes ?

Physiological foundations
System design Objectives System overview DAU CSU System implementation Future improvements Project work summary System demonstration

Connection management Data processing Visualization Data recording

Central System Unit

Connection Connection Manager Manager Data Analysis Visualization Module Data Logger

Bluetooth

Access verification System maintenance

Uses buffered thread-safe queues and producer-consumer scheme

register at
Data consumer

Data producer

Data queue

Data Data Processing consumer & consumer producer

Data queue

Data consumer Data Data Processing consumer & consumer producer

Data queue

Connection Manager Operator manager (OM)

...

OM

OM

Alerts

Separated physiological data streams

Data Analysis

Visualization Module
Processed data

Data Logger

Recorded (off-line) data

Operator Data Manager

Bluetooth

... ...

Bluetooth Connection Manager

Bluetooth

Data demultiplexer Initializes Bluetooth Data producer queues modules Inquires for new Separated Raw Data Operator Buffer range streams data devices in Manager Alert buffer Establishes Alerts connections ... Supports operator ... roaming Feeds Operator Raw Data Operator Buffer Manager with Manager Alerts Alert buffer physiological data

Custom data analyzers - decision tree based classifiers Example: Operator gesture recognition
(simplified) Vertical Acceleration variance value high Horizontal Acceleration variance high

N
Horizontal Acceleration variance high

N
YES

Y
NO

No answer

Motivation What is BlueEyes ?

Physiological foundations
System design Objectives System overview DAU CSU System implementation
Attention level (Lva)

Saccade detector and conscious brain involvement level calculation

Eye movement velocity 250 200 150 100 50 0 -50 -100 -150 -200 -250 0 8000
saccade

Raw eye movement data

Velocity

Differentiation

16000
Time [ms]

24000

32000

Visual attention level

60 50 40 30 20 10 0 0 8000 16000 Time [ms] 24000 32000

Future improvements Project work summary System demonstration

Conscious brain involvement level

 Enable

Superviser to analyze operators duty Voice deliver (of operators) through Voice Data Acquisition Apart from Data Logger - Video Logger _ Handle Camera

Motivation What is BlueEyes ?

Physiological foundations
System design Objectives System overview DAU CSU System implementation Future improvements Project work summary System demonstration

Raw and processed data visualization using: VU-meters pie-charts time series audio and video data playback

Motivation What is BlueEyes ?

Physiological foundations
System design System DAU CSU Future improvements Project work summary
implementation

System demonstration

Standard MDI user interface Custom-built Bluetooth communication routines Intel USB camera ID card programmer Developed using Visual C++ 6.0 Deployment environment: Windows 2000, MS SQL Server

Motivation What is BlueEyes ?

Only

Physiological foundations
System design Objectives System overview DAU CSU

System implementation Future improvements Project work summary

registered mobile devices can connect to the system Bluetooth connection authentication Access rights restrictions Personal and physiological data encryption

System demonstration

DAU small CMOS camera to monitor the Physiological foundations operators point of gaze System design single PCB System implementation low voltage ICs - LiIO batteries power System
Motivation What is BlueEyes ?

demonstration

Future improvements

Project work summary

CSU data mining algorithms advanced database encryption using e.g. AES algorithm

Motivation What is BlueEyes ?

Physiological foundations
System design System implementation

Future improvements
Project work summary System demonstration

Prevention from dangerous incidents Minimization of ecological consequences financial loss a threat to a human life

The reconstruction of the course of operators work

Motivation What is BlueEyes ?

Physiological foundations
System design System implementation

System demonstration
Future improvements Project work summary

BlueEyes
Human-Operator Monitoring System