A Study on the Effect of Lighting

Conditions on the Accuracy of

Heart-Rate estimation From
Facial Video.
Vishay Raina – SR No 14860
Guide: Dr Prasanta Kumar Ghosh
 Introduction
• Heart rate measurement.
• Contact based vs Remote.
• Photoplethysmography (PPG) is an optical technique for detecting
microvascular blood volume changes in tissues.
• Measures from microvascular tissue bed beneath the skin, in which
there are blood volume changes due to the pulsatile nature of the
circulatory system.
• Light source illuminates the tissue, and the photodetector senses the
small variations in reflected or transmitted light intensity associated
with changes in the blood volume in tissues.
 Introduction
• The fundamental principle of PPG relies on the differences in
sensitivity of different optical wavelengths for blood and other tissue
components.
• When the heart pumps blood to the body, the amount of blood that
reaches the capillaries in the skin surface increases, resulting in more
light absorption.
• The blood then travels back to the heart through the venous network,
leading to a decrease of blood volume in the capillaries and less light
absorption.
• Transmission mode vs Reflection mode.
 Introduction
• Limitations: Spot measurement, Contact
Measurement, Motion artifact corruption.
• Imaging Photoplethysmography (iPPG)
• Illuminate the specific tissue with a light
source, and then, measure the light leaving
the tissue with an imaging sensor.
• Reduces the physical restrictions and
cabling associated with patient monitoring
• Avoids the deformation of the arterial wall.
Fig : Schematic setup of an IPPG
system which includes a light source
and a camera.
 EFFECT OF LIGHTING CONDITION ON HR
ESTIMATION .
• The most popular choices were found to be ambient light and white
LEDs.
• A dual wavelength array of LED light source was adopted for pulse
oximetry and HR measurements by Humphreys et al.
• Real life scenarios:
• White light of variable intensity.
• Variability in orientation of the subject's face.
• Prakash et al. Studied five different illumination intensities based on
positioning of the subject with respect to a window (light source).
 EFFECT OF LIGHTING CONDITION ON HR
ESTIMATION
• In our recorded data set two different LED lights of variable light
intensity have been used, and were placed towards the subject's face
at an angle of 48 degrees.
• Videos have been recorded in different configurations that simulate
real life scenarios.
• The light conditions were varied strategically:
1. Subjects were not exposed to the LED light.
2. Subjects faces were partly lit by the LED light.
3. Varying the difference in the intensity of the over exposed and the
under exposed region.
4. Both the sides of the subjects were over exposed to the LED lights.
 Equipment Used: Lights
• Two lights, both containing an array of 8x2 white light LEDs were
used.
• Each light had different modes of operation tabulated as:

Index Mode Intensity values (lux)

Off 0 38
Low 1 45
Medium 2 90
High 3 135
 Equipment Used: Lights
• The different light configurations used are as displayed in the figure
below:
 Equipment Used: Cameras & Oximeter
Cameras Type Resolution Captures
Asus Phone Camera 3840 x 2160 Face
Lenovo Phone Camera 1920 x 1080 Face
Sony Digital Camera 1280 x 720 Face
Samsung Phone Camera 1920 x 1080 Oximeter
• All the videos were recorded at a
sampling rate of 30 fps.
• A commercial pulse oximetry sensor
which is attached to the finger tips of the
subject was used to determine the true
Heart Rate values.
 Recording Setup
• The three cameras (Asus,
Lenovo, Sony) were placed along
the 1 ft or 1.5 ft arc on a table at
the angles 33 degrees from the
center line.
• The light sources were placed
1.3 ft distance in front of the
subject and 45 cm to either side
of the center line at an angle of
48 degrees.
• The oximeter was fixed on the
table towards the right side of
the subject
 ANNOTATIONS
• To get the true heart rate integer values the following process was
followed:
• The image frames were read from the video of the oximeter.
• The portion of the image with HR digits was extracted.
• Segregate each digit and feed into a CNN trained to recognize digits
from 0 to 9.
• Combine the outputs to obtain the true heart rate.
 ANNOTATIONS
• Training Data:
• Training data was generated using the videos from a different dataset of
the same Oximetry device in the following way:
• Choose a rectangle around the HR values in the first frame of the video.
• Manually annotate the HR values displayed inside the rectangle.
• For each frame in the video find the Frobenius norm of the difference
between the intensity values of two adjacent frames.
• Plot these difference values and choose a threshold determining where
there is a change in the heart rate values between two adjacent frames.
• Manually input the value of heart rate in case of such changes, else use the
previous values of the heart rate.
 ANNOTATIONS
• Convolutional Neural Network:

• The first convolution Layer has 32 filters and a kernel size of 5, the
activation function used is ReLU.
 ANNOTATIONS
• The second convolution Layer has 64 filters and a kernel size of 3
and the activation function used is ReLU.
• Both the max pooling layers perform down-sampling with strides of
2 and kernel size of 2.
• The data is flattened to a 1-D vector for the fully connected layer
(fc1). Fully connected layer fc1 has 1024 neurons.
• The output of fc1 is passed to a 10-way softmax, which produces a
probability distribution over ten labels.
• A Dropout rate of 0.75 was used, the learning rate was kept at
0.001, batch size of 128 images.
• Accuracy of 98.43% was achieved on a test set of 10000 images.
 Methodology of IPPG
• ROI Detection
• Facial region is identified from each frame of the video using the built
in implementation of the Viola-Jones face detection algorithm which
uses various Haar- like features.
• The pixel intensity values in the RGB channels were averaged over the
ROI to generate a 3 x N matrix containing the intensity contours
where N is the number of frames in the video.
 Methodology of IPPG
• Independent component analysis technique is applied on the
intensity contour signals to obtain the underlying source signals.
• The JADE (Joint Approximate Diagonalization of Eigenmatrices)
implementation is used to find the demixing matrix W.

• This is used to derive the approximation of the underlying source

signals.
 Methodology of IPPG
• Frequency domain analysis is done on the source signals to find the
frequency values for spectral peaks in the range of 0.75 Hz to 4Hz
(corresponding to 45 beats per minute and 240 beats per minute).
• We choose N such peaks in three different source signals (r), for each
window k.
• The confidence value for the frequency corresponding to the pth
peak, and mth source signal is given by the ratio of the square of the
magnitude of the pth peak in mth source and sum of squares of
magnitude values between 0.75 Hz to 4 Hz.
 Methodology of IPPG
• Prob(true source = r, true peak = p given w_k) is defined as the ratio
of confidence of frequency corresponding to r, p and sum of
confidences of frequencies for all r, p. --(1)
• Then the joint probability of HR in all windows is maximized instead
of maximizing the probability in each window separately.

• --(2)

• Where I_k = (r,p), source and peak pair from which HR is estimated in
k-th window and w_k is an array of frequency locations
corresponding to the N highest peaks in the three source signals
 Methodology of IPPG
• Assuming conditional independence of w_k with respect to I_k, we
get:

--(3)
• Which can be obtained from (1)
• Assuming Markovian independence and applying chain rule we get:

--(4)
 Methodology of IPPG
• This is the probability of the peak value pair changing from I_k-1 in (k-
1)th window to I_k in the kth window.
• It is defined as:
• Where lambda is a hyperparameter.
• Combinig 3 and 4 we get an expression:
•
• The HR sequence is obtained by maximising that expression:
•
• This optimization is solved using dynamic programming is a way
similar to viterbi decoding as proposed by Raseena et al.
References