BIOMETRIC SIGNALS ESTIMATION USING SINGLE
PHOTON CAMERA
Supervisor: Dr. Federica VILLA
Co-Supervisors: Eng. Marco PARACCHINI
Thesis of:
Lorenzo Marchesi Student ID: 859217
Dependability Engineering Innovation for Cyber-Physical Systems (DEIS) is an ICT project funded by the European Commission within the Horizon 2020 program. The DEIS project addresses important and unsolved challenges in assuring dependability within complex Cyber-Physical-Systems (CPS). New paradigms for dependable systems, such as Digital Dependabil-ity Identities (DDIs), will be developed and validated in several use cases, one of which is in the automotive eld. POLIMI SPAD-lab participates to this project developing a Single-Photon Avalanche-Diode (SPAD) camera to perform 3D measurements inside the vehicle and also monitor biometric parameters of the driver, thus developing algorithms for remote Photo Plethysmography (rPPG) measurements. PPG is based on the principle that blood absorbs light more than surrounding tissues so variations in blood volume can be detected considering changes in the light intensity backscattered by the human skin. In rPPG a camera is used to acquire a video, typically of the subject face. Main focuses of this thesis are the estimation of Heart Rate (HR), Heart Rate Variability (HRV) and Respiration Rate (RR) of the drivers using the SPAD camera. Dierent devices were used to validate the results: a commercial RGB camera to measure rPPG with a more standard camera, portable ECG (Faros 180) in order to have a ground truth for heart signals, breaths resistive sensor to measure RR. Sev-eral measurements were performed to optimize the experimental setup. The rst acquisitions with AC illumination (from a neon lamp) returned very low quality results. Thus further measurements have been acquired with an active DC illuminator and optical lters in front of the SPAD camera, with the purpose to compare the contribution of dierent wavelengths on the signal modulation. Measurements were performed with 10 optical lters from 400 nm to 850 nm (40 nm lter width, 50 nm steps) on ve volunteers. The acquisitions lasted 10 minutes with the subject still in front of the SPAD camera, wearing a portable ECG for validation. Heart rate, tachogram and its spectrum were computed in post processing and compared to the same parameters estimated from the ECG signal. The highest accuracy
(camera SPAD) per eettuare immagini 3D all'interno del veicolo e monitorare le condizioni di salute del guidatore, sviluppando algoritmi per misure di foto pletismogra a da remoto (rPPG). La PPG si basa sull'assorbimento di luce, maggiore nel sangue ossigenato rispetto ai tessuti circostanti. Le variazioni di volume sanguigno possono essere misurate considerando i cambiamenti dell'intensità della luce ri essa dalla pelle. Nella foto pletismogra a da remoto (rPPG) tipicamente si inquadra il volto del soggetto. Gli obiettivi di questa tesi sono la stima della frequenza cardiaca, della variabilità cardiaca, e della frequenza respiratoria del guidatore utilizzando la camera a singolo fotone. Per validare i risultati ottenuti sono stati utilizzati diversi dispositivi: una camera RGB commerciale, per avere un confronto sulle misure ottenute con una tipologia camera più standard, un elettrocardiografo portatile (FAROS 180), per avere un gold standard di confronto per il segnale cardiaco e un sensore resistivo per misurare il respiro. Sono state eettuate diverse misure per arrivare a de nire un setup sperimentale ottimizzato. Le prime acquisizioni con illuminazione AC (data da lampade al neon) hanno fornito scarsi risultati, per questo motivo le successive acquisizioni sono state eettuate con un illuminatore attivo alimentato in DC e dei ltri ottici sull'obiettivo della camera SPAD, con l'obbiettivo di confrontare i contributi delle diverse lunghezze d'onda sul segnale cardiaco estratto. Le misure sono state eettuate utilizzando 10 ltri ottici tra 400 nm e 850 nm (con 40 nm di larghezza spettrale), con un passo di 50 nm, su cinque volontari. La durata
calcolati frequenza cardiaca, variabilità cardiaca e relativo spettro e sono stati confrontati con i rispettivi dati estratti dal tracciato ECG. La migliore accuratezza è stata ottenuta dal segnale con lunghezza d'onda centrata a 550 nm e 850 nm (consigliata per motivi di eye-safety). Ulteriori acquisizioni di 10 minuti sono state eettuate su 3 soggetti a 550 nm, insieme a misure di respiro e al video della camera RGB, ottenendo elevata accuratezza per frequenza cardiaca, variabilità cardiaca e frequenza respiratoria sia su nestre temporali di breve durata (30 secondi) sia di lunga durata (10 minuti). Sono stati sviluppati e testati con misure sperimentali due illuminatori, uno a 550 nm e uno a 850 nm. Le ultime misure sono state eettuate all'interno di un'automobile installando la camera SPAD e gli illuminatori all'interno dell'abitacolo. Problematiche relative a artefatti da movimento e modulazione della luce naturale non sono state arontate in dettaglio e saranno oggetto di sviluppi futuri. In particolare, algoritmi di riconoscimento del volto e tracking in aggiunta a ltri adattivi possono essere sviluppati per risolvere questi problemi, aprendo questa tecnologia a nuove possibilità nel mondo automotive.
1 Introduction 1
1.1 Remote photoplethysmography . . . 2
1.2 Goal de nition . . . 2
1.3 Biometric signal estimation using SPAD camera . . . 3
1.3.1 Algorithms and processing . . . 3
1.4 Work structure . . . 4
2 State of the art 5 2.1 Heart rate monitoring . . . 5
2.2 Heart Rate Variability analysis . . . 6
2.3 Remote Photopletismography . . . 8
2.3.1 Setup . . . 8
2.3.2 Algorithms . . . 9
2.3.3 Results and problems . . . 14
3 Materials 16 3.1 Single Photon Avalanche Diode (SPAD) camera . . . 16
3.1.1 SPAD pixel performance . . . 17
3.1.2 64 x 32 SPAD camera . . . 17
3.2 RGB camera . . . 18
3.3 ECG . . . 21
3.4 Respiration measurement device . . . 21
3.5 Acive illumination . . . 22
4 Methods and setups 25 4.1 First attempts . . . 25
4.4.3 Tachogram calculation . . . 46
4.4.4 Respiration rate from tachogram . . . 47
4.5 Led illuminator . . . 47
4.6 Acquisitions in the vehicle . . . 48
4.7 Adaptive ltering . . . 49
5 Results 51 5.1 AC active illumination and DC powered light setup . . . 51
5.2 Wavelength selection . . . 51
5.2.1 Single beat detection . . . 53
5.2.2 Heart rate detection . . . 60
5.2.3 Tachogram calculation . . . 60
5.2.4 HF/LF estimation . . . 62
5.3 Results in the nal setup . . . 68
5.3.1 Heart rate comparison . . . 69
5.3.2 Tachogram error calculation . . . 69
5.3.3 Respiration rate measurement accuracy . . . 71
5.4 Led illuminator . . . 73
5.5 Acquisitions in the vehicle . . . 74
5.5.1 550 nm and 850 nm active illuminator . . . 74
6 Conclusions 79 6.1 Achieved goals . . . 79
2.5 Schematic representation of dierent methods for ROI detection. . . 10
2.6 Left: Bayer lter structure. Right: Bayer lter spectrum. . . 11
2.7 Example of ICA algorithm. . . 12
2.8 Eect of ltering on the signals and Heart Rate estimation. . . 13
3.1 Preliminary 32 x 32 SPAD camera developed for the DEIS project: boards and chip (upper panel) and housing with C-mount (lower panel). . . 19
3.2 64 x 32 SPAD camera, with the USB 2.0 link and two SMA connectors for synchronization with other instruments. . . 20
3.3 acA1920-48gc Basler camera. . . 20
3.4 Left: Faros 180. Right: Faros electrodes positioning. . . 21
3.5 Upper panel: respiration measurement device schematic. Lower panel: respi-ration measurement device. . . 23
3.6 Respiration measurement chart. . . 24
3.7 Upper panel: schematic of the active illuminator; Lower panel: active illumi-nator PCB. . . 24
4.1 Single frame from a video taken with the SPAD camera.The points marked in black are the vertices of the chosen ROI. . . 27
4.4 Pipeline of the new algorithm. . . 29 4.5 Upper panel: raw signal extracted by forehead pixels; lower panel: bandpass
ltered signal. . . 30 4.6 Power spectrum of the extracted and ltered signal. . . 31 4.7 Emission spectrum of the used incandescent lightbulb powered with DC source. 32 4.8 Single beat detection algorithm. Blue line: SPAD signal processing; Red
line:ECG signal processing; Green line: elaboration using both signals. . . 33 4.9 Piece of synchronized SPAD signal and ECG. . . 34 4.10 Segmentation algorithm example. Blue: rPPG signal. Red, green, black: QRS
time positions in the three steps. . . 35 4.11 Gray: all detected beats in the pulse wave. Blue: average pulse wave beat.
Red: standard deviation of all the beats in the pulse wave signal. Upper panel shows an example of optimal beat shape, while lower panel shows discarded shape. . . 36 4.12 Red: power spectral density of ECG signal; Blue: power spectral density of
pulse wave extracted from SPAD camera. . . 37 4.13 Example of actions of the developed algorithm to extract the tachogram. . . . 39 4.14 Algorithm to extract the tachogram. . . 40 4.15 Final re nement of the tachogram. . . 42 4.16 Algorithm to improve the extracted tachogram. . . 43 4.17 Red: tachogram extracted from the ECG track; Blu: tachogram extracted from
SPAD camera video. . . 44 4.18 Red: PSD calculated from ECG tachogram; Blue: PSD calculated from rPPG
tachogram. . . 45 4.19 Red: tachogram calculated from the ECG; Blue: tachogram calculated from
the SPAD camera; Green: tachogram calculated from the RGB camera . . . . 46 4.20 Red: PSD calculated from ECG tachogram; Blue: PSD calculated from SPAD
tachogram; Green: PSD calculated from ECG tachogram; Black: PSD calcu-lated from respiration data. . . 47 4.21 SPAD camera and active illuminator mounted inside the cockpit. . . 48 4.22 Realistic setup. . . 49
5.7 Subject 5: single beat detected in acquisitions with every wavelength. . . 58 5.8 Average standard deviation for each wavelength. . . 59 5.9 Example of heart rate result. . . 61 5.10 Average root mean square error in tachogram calculation for each wavelength
computed with errors in table 5.3. . . 62 5.11 Example of tachogram result. Blue: tachogram extracted by pulse wave; Red:
tachogram calculated by ECG track. . . 63 5.12 Example of tachogram result. Upper panel shows an example of high accuracy
result; Lower panel show an examle of low accuracy result. . . 64 5.13 Example of tachogram spectra. Blue: spectra extracted by pulse wave; Red:
spectra calculated by ECG track. . . 66 5.14 Average root squared error for each wavelength. Orange: LF components; Blue:
HF components. . . 68 5.15 Example of heart rate detection in a one-minute window using RGB and SPAD
camera.It is clearly visible that both cameras detected 49 bpm in this window, that exactly match the heart rate calculated with the ECG track. . . 69 5.16 Example of tachogram calculated from signal extracted by SPAD camera (blue),
RGB camera (green) and Faros 180 (red). . . 70 5.17 Example of respiration rate calculation, computed as the FFT of the tachogram
of the signals from the three devices compared with the FFT of the signal measured with the respiration measurement device. . . 71
green lines are results obtained using RGB camera. . . 72 5.19 Results of tachogram calculation using manufactured illumination at 550 nm
and 850 nm. . . 73 5.20 PSD of the tachograms of gure 5.19. All values in the graphs are expressed
in s2. . . . 74
5.21 Upper panel: lens with optical length equal to 8 mm; lower panel: lens with optical length equal to 16 mm. . . 75 5.22 Example of signal with motion and light noises. . . 76 5.23 Final re nement of the tachogram. . . 76 5.24 Example of tachogram computed from the rPPG signal (blue) and ECG track
(red). . . 77 5.25 Exmple of respiration rate calculated using SPAD camera (blue), ECG (red)
5.5 Average errors in determination of heart rate in one-minute windows. Measure-ment unit is intended in "average heart beat", i.e. an error equal to 1 means a frequency of 61bpm instead of 60bmp. . . 70 5.6 Mean square error between tachogram extracted from cameras and ECG. . . 70 5.7 Average errors in respiration rate calculation. (Values of circles in gure 5.18).
Errors calculated as mean square errors between the measurments taken with the breathing sensor and the three devices. . . 72 6.1 Errors obtained using spad camera with respect to gold standards (ECG and
respiration measurement device). . . 80
HR Heart Rate
HRV Heart Rate Variability
PPG Photopletysmography
rPPG Remote Photopletysmography
NTC Negative Temperature Coe cient
PCB Printed Circuit Board
LED Light Emitting Diode
CCD Charge Coupled Device
ROI Region Of Interest
ZCA Zero-phase Component Analysis
ICA Independent Component Analysis
DFT Discrete Fourier Transform
FF Fill Factor
NTC Negative Temperature Coe cient
bpm beats per minute
Introduction
Dependability Engineering Innovation for Cyber-Physical Systems (DEIS) is an ICT project funded by the European Commission within the Horizon 2020 program.
The project is coordinated by AVL, worldwide leader in the development of powertrain sys-tems with internal combustion engines, and consists of ten partners, four industrial companies (Siemens, General Motors, Ideas & Motion, Portable Medical Technology), three Universities (Politecnico di Milano, University of Hull, University of York), and two research centers (Fraunhofer-Gesellschaft, Dundalk Institute of Technology), from dierent European coun-tries (Italy, Austria, Germany, United Kingdom, Ireland).
The DEIS project addresses important and unsolved challenges in assuring dependability within complex Cyber-Physical-Systems (CPS), by developing technologies that form a sci-ence of dependable system integration. These systems are receiving broader interests and attention since they harbor the potential for vast economic and societal impacts in domains such as mobility, home automation and delivery of health. At the same time, if such systems fail, they may harm people and lead to temporary collapse of important infrastructures with catastrophic results for industry and society. Thus, ensuring the dependability of such CPS systems is the key to unlocking their full potential and enabling European industries to de-velop con dently business models that will nurture their societal uptake.
New paradigms for dependable systems, such as Digital Dependability Identities (DDIs), will be developed and validated in three use cases: Automotive, with the development of a stand-alone system for intelligent physiological parameter monitoring and for enhancing advanced driver simulators for the evaluation of automated driving functions; Railway, with the val-idation of dependable exchange of information between components and subsystems within complex environments, such as heterogeneous railway systems; Healthcare, for improving
(RR).
1.1 Remote photoplethysmography
Nowadays algorithms performing rPPG (remote photoplethysmography) are composed by two main procedures: face detection and tracking and signal extraction and processing. This work focuses on the latter, thus algorithms to extract biological signals are developed. Moreover, particular attention is paid in the design of a setup that helps in the extraction of these signals.
To the extent of our knowledge, no studies developed a rPPG system using a SPAD camera. This is the main focus of this study: perform a remote photoplethysmography using a SPAD camera chosen for its capability to work in low light conditions and also perform 3D measurements, which can be useful to monitor the movements of the driver within the vehicle.
1.2 Goal de nition
Recent studies showed the possibility to determine HR by recording a video of the face of a subject. This procedure allows to monitor subjects remotely, thus with contactless and non-invasive measures. All these studies used commercial RGB cameras, obtaining good results in the determination of HR. Few studies [1, 2, 3] tried to extract the HRV starting from the rPPG.
Since the DEIS project aim was to both make 3D measurement and subject monitoring inside the cockpit, it was decided to develop algorithms that allows the SPAD camera to measure biometric parameters such as HR, HRV and RR.
1.3 Biometric signal estimation using SPAD camera
In order to perform and validate biometric measurements with a PSAD camera, a gold stan-dard was needed. It was decided to use a portable ECG device in order to have a ground truth for the HR and HRV determination. Moreover, in order to calculate RR, a device measuring this parameter was needed. It was decided to design and manufacture this device.
1.3.1 Algorithms and processing
Developed algorithms begin with a Region Of Interest (ROI) selection on the images recorded by the camera. As stated in literature [4, 5, 6], and veri ed in this work, the regions containing the highest amount of information about heart activities are forehead,cheeks and nose, but cheek can be covered by beard and nose is present in too few pixel, thus the ROI was a rectangle containing subject forehead pixels. In the performed measurements, the subject sits still in front of the camera with the arti cial illumination directed on its face. The values of the pixels inside the ROI were averaged resulting in a pulse wave. This was the starting signal that was processed to nd HR, HRV and RR.
The signal was corrupted by noise thus it must be cleaned using lters, in particular a bandpass lter between 0.4 and 4 Hz. By performing an FFT on this signal the heart rate was obtained. Studies were conducted to evaluate if considering limited spectral ranges the quality of the signal increased, so a study was made to determine which wavelengths returned the highest quality signal. In order to state the choice of the wavelengths four parameters were considered: single beat shape, heart rate detection accuracy, tachogram calculation accuracy and heart rate variability spectrum accuracy. An exclusion principle was followed in order to determine the best wavelengths.
Single beats were obtained dividing the rPPG signal in windows estimated starting from the ECG signal. This rst test excluded 600nm wavelength, as it returned the worst results in terms of shape of the single beats and standard deviation with respect to the median beat. Heart rate detection was obtained performing a FFT on the ltered pulse wave and it was noticed that 600 nm and 650 nm provided the worst results.
Tachogram calculation needed more complex algorithms. These algorithms are composed by two main procedures: one calculating a raw tachogram, and the other re ning it. The spectrum of the tachogram was obtained performing a FFT on the tachogram. Results showed that wavelengths centered in 550 nm and 850 nm returned the highest quality signals. For this
This thesis is organized as follow: section 2 describes previous works on remote photoplethys-mography and de nes heart rate variability and tachogram.
In section 3 all designed and used devices are explained in details; a detailed description of the SPAD camera, RGB camera, ECG and respiration measurement device are provided. Section 4 describes in detail all algorithms developed and the followed methodologies to ob-tain the nal results.
Results are reported in section 5 showing the results of each setup described in section 4. Conclusions resume all aims and relative results, showing which will be the future develop-ments.
State of the art
The importance of health monitoring is rapidly growing since is now possible to store and analyze big amount of data. This eld is evolving in particular in the area of non-invasive biometric signal measurements such as heart rate, blood pressure and respiration rate. The aim of collecting these data is to monitor a subject in order to prevent any pathology or illness without invasive procedures. Technology development is nowadays focusing on biometric signal extraction using videocameras. There are several smartphones applications that can detect blood pulse using the equipped LED and camera . Recent studies focus on the analysis of heart rate from skin using videocameras, showing that it is possible to perform a remote-photoplethysmography under controlled conditions. The correct extraction of these data could lead to deeper analisys such as the calculation of a tachogram, that represents the time intervals between successive heart-beats, as explained in section 2.2, and of respiration rate.
2.1 Heart rate monitoring
Many studies focus on the extraction of heart rate using videocameras [4, 6, 7, 8, 9, 3, 1, 10]. This is possible since light is re ected in dierent ways depending on the amount of blood owing in vessels beneath the skin. In particular, by considering dierent regions of interests (ROIs) on the skin of a subject it is possible to see a modulation on the light with frequencies corresponding to heart rate. Actually what is recorded by the camera is the pulse rate. The camera can detect the changes in blood ow linked to the contraction of the heart. This
Figure 2.1: Skin re ection of light.
signal is dierent from the electrocardiogram since the sources of the two signals are physically dierent: the former is mechanical while the latter is electrical. Pulse rate and heart rate are not necessarily synchronized due to mechanical delays, but they show the same trend so pressure wave frequency corresponds to the frequency of beating heart, as showed in gure 2.2. Pulse rate and heart rate have harmonic components in the low frequency, physiological values at rest are between 0.7 to 2 Hz, i.e. 42 to 120 beats per minute (bpm). The cardiac frequency varies under the control of autonomic nervous system. This control is visible in heart rate variability (HRV).
2.2 Heart Rate Variability analysis
Deeper analysis of the Heart rate lead to Heart Rate Variability (HRV). HRV is the analysis of the inter-beat-interval, that is the time dierence between two consecutive R waves in the ECG.
The importance of HVR in medical eld is rapidly growing in the last years since its modula-tions are due to autonomic nervous system. This makes HRV an important health indicator useful to prevent dierent diseases such as diabetes, iper-tension, cardiac brillation[11, 12]. HRV could be represented using a tachogram, which is a chart reporting time on the x axes and time interval between two consecutive R waves on the y axes, see gure 2.3. The spectrum
Figure 2.2: ECG signal (a) and PPG signal (b): the image show the mechanical delay introduced by circulatory system, but the time distance between two peaks in ECG and PPG is the same.
of this signal is of the utmost importance in determining the diseases mentioned above. The spectrum of the thacogram presents two dierent main components that are commonly called Low Frequency component (LF) and High Frequency component (HF). The ratio between this two quantities is a measure of the simpatho-vagal balance, or rather gives a quantita-tive information about the functioning and the activation of the autonomic nervous system. These teo components are between precise ranges of frequency: LF is between 0.04 to 0.15 Hz, while HF is between 0.15 to 0.4 Hz. The LF component is considered a marker of sympa-thetic modulation, but actually include also vagal in uences, while HF is strictly connected to respiration. The peak of the HF component in a normal subject at rest correspond to the respiration frequency.
In conclusion, performing a spectral analysis of the tachogram leads to the following informa-tion: Heart Rate, LF/HF balance and respiration rate.
The goal of most recent studies is to obtain the tachogram by performing a remote-photoplethismography[6].
Figure 2.3: Upper panel: rapresentation of a tachogram. Lower panel: non-parametric and parametric spectrum of the tachogram.
Figure 2.4: Typical recording setup.
2.3 Remote Photopletismography
Verkruysse et al. [4, 13, 1, 10] were pioneers in showing that a video captured with a com-mon RGB camera is enough to obtain a plethysmographic signal whence measuring HR and respiration rate.
2.3.1 Setup
Typical setups for validating remote-photoplethysmography involve the use of a low-cost RGB camera, and devices used to obtained ground truth values for HR, HRV and respiration rate, such as electrocardiograph, pulse-oximeter, and devices to measure respiration.
In these experiments the face of the subject is recorded. The subject is asked to stand still in front of the camera. Dierent subject positions have been explored in dierent works: de
Haan and Poh [13, 10] for example, asked the subject to sit still in front of the camera at a distance of 1-2 meters; Iozzia [9] considered a rst period of rest lying horizontally and a second period where the subject was kept vertically, thus seeing the modulation of the sympatho-vagal balance; J. Moreno [6] investigated the dierences between the supine position and the sitting position.
In the majority of these setups the distance kept between the subject and the camera varies from 30 cm to 1 meter and the illumination used is commonly ambient light, with exception like DeHann that used professional studio illumination [10] or J. Moreno [6] that used a low amplitude uctuation lamp.
Results in the exctraction of parameters of interest were compared with signals extracted by dierent devices. Heart rate was commonly recorded with an electrocardiograph or a pulse-oximeter, while the respiration rate was measured with inductive band [14] or thermistor at external naris [15].
Particular attention is paid to the choice of the camera and the camera settings like frame rate and resolution.
The choice of the camera is critical as long as it is a single device used to extract all the biological signals mentioned above. Cameras used in literature are commonly CCD (Charge Coupled Device) cameras, some studies use webcam integrated in laptop [13, 16], while others record video using compact-cameras or giga-Ethernet-cameras [9]. The resolution of these devices varies around 640x480 pixels, but is enough to extract the signal. All these cameras are RGB cameras so the output is composed by 3 channels, red, green and blue, and the depth resolution is 8-bit per channel. In studies where a compact camera is used, acquired images are stored as raw and analyzed o ine. This allows to avoid compression and the connected loss of information from the single image. Particular attention is commonly paid to acquisition frequency but dierent works provide dierent values: Poh considers 15 fps [13], while others acquire at 20 to 60 fps.
Once the setup has been established the aim is to extract the biological signals from the video and compare them to signals coming from the other devices.
2.3.2 Algorithms
Several algorithms were developed to extract heart rate, heart rate variability and respiration rate. As mentioned in section 2.1, the waveform of the signal extracted from a camera is
Figure 2.5: Schematic representation of dierent methods for ROI detection.
completely dierent from the one extracted from a PPG or an ECG, but there is a strong correlation in the frequencies and in the relative time position of particular features like peaks or zero-crossing. The goodness of the described setups is based on these correlations.
2.3.2.1 ROI detection
The video focuses on the face of the subject and a Region Of Interest (ROI) is selected in each frame. As long as the subject is asked to avoid head movement the selection of the ROI can theoretically take place only in one frame.
There are several ways to determine this ROI. The easiest one considers a region by manually choosing the pixels of the image corresponding to the skin of the subject [7, 3]. In this kind of choice tipical ROIs are rectangles and the selected portion of the face are forehead and cheeks. Other approaches are based on face recognition and tracking [17, 13]. In these cases two dierent methodologies were developed: one uses an algorithm of face detection and recognises the forhead using particular ratios of the dimension of the face, while novel works focuses on the detection of skin pixels [8, 17]. In this second case the considered surface is wider, but it shows low performances in presence of beard or glasses. Once the ROI is selected, the signal extracted is given by the mean of the value of all the pixels in the region, thus obtaining a matrix with 3 rows, one for each channel, and n columns, where n corresponds to the number of the frames.
Figure 2.6: Left: Bayer lter structure. Right: Bayer lter spectrum.
2.3.2.2 Channel selection
The output given by the camera is divided in 3 channels: red, green and blue. This is due to the Bayer lter placed on the sensors of the camera. The Bayer lter is a grid composed by optical bandpass lters centered in the wavelength of red (600 nm), green (550 nm), and blue (450 nm) as showed in gure 2.6. In this way each pixel can collect only the intensity of one single color and the information is typically converted in 8 bit resolution.
Once the mentioned matrix is obtained, attention is paid to the choice of the channel with more information. Dierent approaches are used: some consider single channel, while other consider a combination of the dierent channels, trying to maximize the information hidden in the matrix. The easiest method explained in literature takes into account the information embedded only in the green channel. J. Moreno et al. [6] performs a decomposition of the 3 channels and extracts the signal considering only the G channel. They showed that the cardiac signal is present in the 3 signals, with the G component having the highest amplitude, so the G component was chosen in theirs study to obtain the cardiac information. Also V. Rouast [7] arrived to the conclusion that G channel contains enough information and also recommend the use of the single G channel in order to reduce computational costs and to implement online analysis.
A completely dierent approach was used by de Haan [10], who considered all the channels in order to remove movement artifacts; they found out that given the pulsatility as a function of wavelength exhibits a strong peak in green and the dips in red, so a ratio of normalized green and red would make a motion robust pulse signal. Further development lead to the usage of all the three channels and in particular their dierences in order to obtain a chrominance signal. This returned good signal and robustness to motion artifacts. A similar approach was followed by Iozzia [9] that considered the chrominance model, but also performed zero-phase component
analysis (ZCA) and a combination between chrominance model and independent component analysis (ICA) on the three channels to extract a robust signal corresponding to hear rate. Also Poh [13] used ICA in its studies. ICA and ZCA works similarly: given independent signals these algorithms detect all the components that are present in all channels and clearly separates them. In these studies the inputs are the three channel and these algorithms are capable to reject motion and noise components and to extract he pulse wave.
Once the channel or the combination of channels is selected the extraction of HR, HRV and respiration rate can be developed.
2.3.2.3 Signal processing
After extracting the signal, the majority of the proposed methods apply some data ltering techniques in order to remove noises due to electronic interference and quick movements. As long as these lters usually are band-pass lters, also slow components are removed from the signal; these components are due to slow movements of the subject during acquisition, but also to slow and small variations in illumination.
The signal searched has frequency components between 0.4 and 2 Hz so the band-pass lter showed in most of the studies has passband between 0.4 and 4 Hz to avoid introduction of processing artifacts. Most recent publication developed an adaptive bandpass lters, which dynamically changes the cut-o frequencies based on previously estimated HR [2].
Once the signal is estimated the analysis proceed to nding the signal frequency components. To achieve this result a Discrete Fourier Transform (DFT) is performed, leading to nd the peak corresponding to HR, as shown in gure 2.8.
he considered that if the change from one RR interval to the next exceeded the interquartile range of the dierentiated RR time series 10-fold, an artifact was present. The artifact was then classi ed as a missing beat (FN), extra beat or ectopic-like beat (FP). For a missing beat, the corrected RR interval was obtained by adding as many consecutive RR intervals as necessary to obtain a value close to the mean of the previous 10 RR intervals. An extra beat was split into as many RR intervals as necessary with an equal value, so as to be as close as possible to the mean of the previous ten RR intervals. An ectopic-like beat was substituted by 2 equal RR intervals, corresponding to the mean of the 2 RR intervals involved in the ectopic like beat. To perform the calculation on the parameters extracted from HRV, such as LF and HF, explained in section 2.2, he considered the green channel singularly. In order to improve the temporal resolution on the detection of the beat position the signal was resampled from 30 Hz to 1 kHz using a cubic spline. The obtained signal is then bandpass ltered and compared with a threshold equal to 0.8 times its standard deviation. The local maxima in the resulting signal were used as estimates of the beat position. The RR series was obtained by dierentiation of the beat positions.
Once the RR series is obtained, the FFT of this signal provides the HF and LF components. Considering the frequency band between 0.15 and 0.3 Hz, a peak corresponding to respiration rate is visible [18].
2.3.3 Results and problems
All the studies compared the obtained results with a signal extracted using medical devices such as ECG or contact PPG sensors. Tasli [1] underlined that after detrending and ltering operations on the video signal, the error on the estimation of HR is around 3% and stated that the main limitation of his method is observed under poor lighting conditions. De Hann [10] demonstrated that rPPG provided pulse rate in 92% good agreement with a contact PPG
sensor. De Hann encountered problems in the choice of illumination and in motion artifacts that ruined the signal. For what concern the HRV calculation, Moreno [6] explained that the main encountered problems were facial movements and illumination changes.
In all the considered studies the camera taken into account for the recording of the face of the subject was a RGB camera. All the studies concluded that the green channel contains the highest amount of information related to heart rate, but motion related problems lead to the usage of red and blue channel to remove artifacts. To the extent of our knowledge no study has ever been done with the aim of selecting the best wavelength for rPPG, and this is one of the goals of the presented study. Moreover, no one made experiments with low lighting condition, this probably due to the low performances of common RGB cameras. Another aim of this study is also to develop improved algorithms for HR and HRV extraction with lower error and increase accuracy in detection of heart rate and heart rate variability, and also to extract respiration rate from this second feature.
device performing breath measures and an active illuminator to improve the quality of the extracted signals, were designed and manufactured and their working principles are described in detail.
3.1 Single Photon Avalanche Diode (SPAD) camera
In its simplest form, a SPAD is a reverse biased P-N junction that is operated well above its breakdown voltage, dierently from photodiodes and Avalanche PhotoDiodes (APDs), which are instead operated well below and slightly below breakdown voltage, respectively. The excess voltage VEX = VBIAS − Vbreakdown is de ned as excess bias and impacts all SPAD
performance. As soon as a photon impinges on the SPAD active area, the photogenerated electron and hole are accelerated by the high electric eld within the active volume and they can impact against an atom and cause ionization (i.e. the creation of other electron-hole pair). Thanks to the positive feedback loop (being the junction biased above breakdown), the process ignites an avalanche multiplication build-up, leading to a macroscopic, self-sustained current in the milliamp region. A quenching circuit detects the current onset, then it quenches the avalanche process (by lowering the bias below breakdown) and, after a selectable hold-o time, it re-arms the SPAD back to its original above-breakdown bias, thus enabling other photon detections. Such avalanche quenching must be performed as quickly as possible, in
order to limit the total amount of charge owing through the detector, as this quantity is in direct correlation with the very undesired eect of afterpulsing and self-heating. The active area, i.e. the distance of the outermost SPADs, is 4.8mm x 4.8mm, since each chip has a pitch of 150µm, while each SPAD has a diameter of 30µm. The number of pads to properly control the chip is 172 (109 for the two power supplies and 63 for signals), so a suitable FPGA board is needed to operate the imager.
3.1.1 SPAD pixel performance
Non-idealities of SPAD detectors, such as their nite photon detection e ciency (PDE) and not-nil count rate (dark-counting rate noise, DCR) even when no photons impinge on their active area, can play a fundamental role in limiting the signal-to-noise ratio when such sensors are employed as either 2D photon-counting imagers or 3D photon-timing (e.g. Time-of-Flight) distance ranging sensors. Both PDE and DCR increase with excess bias, therefore they must be trade-o when selecting VEX. The POLIMI's 64 x 32 SPAD chip shows a median DCR of
60 cps (counts per second) when measured at 25oC room temperature. This value is still the
present state-of-the-art among Silicon SPADs fabricated in a CMOS process, and still as good as custom-processed SPADs. The DCR can be decreased by lowering the temperature of the chip, e.g. by means of a Peltier cooling stage. Punctual imperfections can cause an increase in local impurity concentration, originating a "hot-pixel", i.e. a SPAD with much higher DCR with respect to the median value. For the reported 64 x 32 SPAD array with 30µm SPADs, the percentage of hot pixels is just 5%, with the highest DCR being 35 kcps, still reasonable for most applications. The PDE together with Fill-Factor (FF) are key parameter when single-photon sensitivity is required: in fact, even if SPAD detectors are in principle able to detect single photons, a not-negligible fraction of the incoming photons are not able to ignite an eective avalanche current multiplication process, either because they are absorbed outside the photon absorbing area or because they do not have enough energy to generate an electron-hole pair, or such a pair does not succeed in igniting the avalanche ionization process. This e ciency strongly depends on the incoming photon wavelength and on the excess bias applied to the detector.
3.1.2 64 x 32 SPAD camera
For the DEIS project, POLIMI developed dedicated electronic boards to properly drive the 64 x 32 SPAD array chip developed in the past projects. The purpose was to ascertain
chip, and nally the top board hosts the SPAD array chip. The camera can be easily connected to a remote PC and can be self-powered through a simple USB 2.0 link. The electronics is able to apply to the SPAD array a selectable bias voltage VBIAS in the range 23-31 V, which
guarantees the possibility to operate SPADs with up to 6 V excess bias, being the SPAD Breakdown voltage Vbreakdown = 25V. The power consumption is always less than 10 W
(when supplied by a 5 V, 2 A external power supply) even in worst case conditions, namely at 50 Mcps count rate per each pixel in photon-counting, well beyond the maximum operating conditions within the DEIS application. Data processing is performed by a commercial board produced by Opal Kelly Incorporated (model XEM6010), featuring USB 2.0 communication link, a Xilinx Spartan 6 FPGA (Field Programmable Gate Array) device, a 128 MiB, 10 Gib/s DDR2 SDRAM memory from Micron. To increase image dimension, POLIMI has developed a camera based on a 64 x 32 pixels image sensor. Each pixel of the sensor comprises a SPAD and three 9-bit counters to count the number of incoming photons. An FPGA board is used to readout and save in a SDRAM the data acquired by the imager and the camera interfaces to the pc through a USB 2.0 link. Two SMA connectors allow to synchronize the camera with other instruments. A picture of the preliminary 64 x 32 camera is shown in gure 3.2
3.2 RGB camera
As shown in Chapter 2, many studies developed systems to extract biometrical signals using RGB cameras. In order to determine and compare the performances of the presented setup with a more standard technology, the use of a RGB camera was necessary. To achieve this goal a Basler GigE RGB (Red Green and Blue color) camera was employed. The model of the chosen camera is acA1920-48gc. This is a microcamera that can reach up to 50 fps with global shutter and a resolution of 1920 px x 1200 px. The sensor type inside the camera is built in
Figure 3.1: Preliminary 32 x 32 SPAD camera developed for the DEIS project: boards and chip (upper panel) and housing with C-mount (lower panel).
Figure 3.2: 64 x 32 SPAD camera, with the USB 2.0 link and two SMA connectors for synchronization with other instruments.
Figure 3.3: acA1920-48gc Basler camera.
CMOS technology. This Basler camera is RGB with a pixel depth of 10 bits. This camera has been chosen for two main reasons: exposure time, frame rate and synchronization are programmable (note that the camera is connected to the computer by means of an ethernet cable), and the lens mount is a C-mount, that is the same of the used SPAD camera, thus making it easer to share the small optics (objectives and lters) for both the cameras. Sensor dimensions are 9.2mm x 5.8mm with pixel size of 4.8µm x 4.8µm, making this camera one with the most similar sensor dimension to the SPAD camera.
Figure 3.4: Left: Faros 180. Right: Faros electrodes positioning.
3.3 ECG
To obtain a reference for the cardiac signal, a portable ECG has been used. In particular the Faros 180◦. This device acquires data from three surface electrodes, placed as shown in gure
3.4, thus giving three ECG traces, one for each derivation. The device also contains triaxial accelerometer, collecting information on acceleration of the device itself in all directions. This function has been used as time reference during the measurements: the device was hit at the beginning of the acquisition in order to obtain a clear peak in accelerator data and consider this point as the start of the acquisition. Faros 180 is programmable by a user interface where acquisition frequency of both accelerometers and electrocardiogram can be selected; in this study a frequency of 250 Hz for ECG and 400 Hz for accelerometers has been chosen.
3.4 Respiration measurement device
Respiration rate measurement is another aim of the presented study and in order to obtain a ground truth to validate the developed algorithm and proposed setup, a non-invasive device for respiration measurement has been designed and manufactured (Figure 3.5). This device is mainly composed by a thermistor that must be positioned under the external naris to measure temperature changes during normal breath; an ampli cation circuit is needed to obtain a stable signal and the acquired data are collected by means of an Arduino interfaced with Matlab. The thermistor chosen is an Negative Temperature Coe cient (NTC) resistor with a resistance at room temperature of 10 kΩ. This kind of resistors decrease rapidly their resistance when temperature is increasing, thus, if positioned under the external naris, the
3.5 Acive illumination
Active illumination is needed to develop a stable and repeatable setup. Natural light varies slowly its intensities due to clouds, time and weather. Conventional lightbulbs and neon provide wide wavelengths spectra, but show problems related to stability. This kind of light is powered by 220V working at 50Hz. This frequency creates aliasing phenomena in the collected data, adding noise to the measured signal. The solution is to use an active illuminator powered by DC current. As it will be explained in section 4.2, attempt were made with LEDs lightbulbs, that solve the problem of frequency aliasing. This kind of illumination provides two main problems: it must be powered with 220V and provides a wavelength spectrum that doesn't match correctly the desired one. Finally it was decided to design a Printed Circuit Board (PCB) to obtain the illumination that perfectly ts the developed setup. As shown in gure 3.7, this circuit is composed by 10 Light Emitting Diodes (LED) and resistances, thus it can be powered with 12 V supply, and it provides the correct intensity for the measurement using the presented setup and allow to select the correct wavelength.
Two illuminators were manufactured, one using LEDs emitting at 550 nm and the other emitting at 850 nm. Tests characterizing illuminators safety were performed using a power-meter. Results showed that the former illuminator positioned at a distance of 50 cm produce an optical radiation power of 100 mW/m2, ten times below the maximum limit of allowed
radiation power, while the latter, produce an optical radiation power equal to the maximum allowed, 20 W/m2, in order to extract a signal of the same amplitude in both cases.
Figure 3.5: Upper panel: respiration measurement device schematic. Lower panel: respiration measure-ment device.
Figure 3.6: Respiration measurement chart.
Methods and setups
Chapter 4 describes the experimental setups used to perform preliminary measurements and the methods and algorithms developed to process the acquired signal and extract the required biometric informations. The main focus was in the selection of the illumination light, which should be stable, in DC and at speci c wavelengths to improve the quality of the measured signal. All the algorithms implemented to extract the signal and provide an estimation of the Heart Rate, Inter Beat Interval, Heart Rate Variability and Respiration rate, are described in details.
4.1 First attempts
The rst step to extract heart rate by using a SPAD camera is to record a video of the subject. The camera records the face of the subject, in accordance with literature setups. As previously explained, the SPAD camera can work at very high frequency leading to large les in terms of memory. Moreover, the high number of frames increase processing time. It was decided to use the provided software to control the camera and select the lowest frame rate, at a frequency of 416 Hz .
In the rst setups neon light was used, the subject sit still in front of the camera and the acquisition lasted one minute. Using Matlab, a rectangular region over the forehead of the subject was selected; an example is shown in gure 4.1, where the four black dots represent the vertices of the rectangle. This selection was performed in the central frame of the whole acquisition and the considered ROI was kept for all the frames. Due to movements of the subject, the forehead could move outside the selected rectangle, introducing noise in the signal. Major movements were visible at the beginning of the acquisition, thus the central
of photons counted in the selected region. By creating an array of these values extracted from each frame and plotting them, it was possible to visually identify the trend in time domain of the photons coming from the forehead of the subject. This was considered from hereon as the Heart pulse wave. With this setup the power source frequency of the light was clearly visible in the signal resulting in an extremely high noise that completely overlay the signal of interest.
The following step was the determination of the Heart Rate and it was achieved by performing a simple FFT algorithm. The results showed that the 50 Hz component (and its harmonic) was the main component of the pulse wave, but a smaller peak was visible around 1.1 Hz, which corresponds to the average heart rate. Another issue were the slow trends of the resulting signal, mainly due to slow movements of the subject and slow light changes. The main problem of this setup was de nitely the light source, that provided the largest noise component. The pipeline of the algorithm is resumed in gure 4.2.
It was decided to change the arti cial illumination and use a DC powered LED strip.
4.2 DC powered light setup
The arti cial illumination in this second setup was composed by 4 LED strip. Each LED was able to emit three dierent colors, red, green and blue, and combine these to obtain white light. Also in this setup the subject sit still in front of the camera and the illumination was positioned behind the camera in order to enlight the face of the subject, as shown in gure 4.3. Recordings lasted one minutes and the acquisition frequency was still 416 Hz. In addition to the previous setup, an optical lter centered in 500 nm (40 nm Full Width at Half Maximum) was mounted on the camera. This choice was made in accordance with literature, where the green channel of the RGB cameras was considered the one containing the highest information
Figure 4.1: Single frame from a video taken with the SPAD camera.The points marked in black are the vertices of the chosen ROI.
Figure 4.3: LED strip setup. On the right SPAD and LED strip light are shown.
ones recorded with the previous setup. Pulse wave modulations were clearly visible in the extracted signals, but the signals needed post processing algorithms to be useful in heart rate calculation.
Once the signal was extracted, it was ltered using a Butterworth bandpass lter implemented with Matlab. The considered band for the ltering was between 0.4 Hz and 4 Hz, that is 24 to 240 beats per minute, so containing the normal resting cardiac frequency. The FFT calculated showed a clear peak at frequencies around 1.1 Hz. A graphical description of this improved algorithm is shown in gure 4.4.
Figure 4.5 shows the extracted signal from a piece of video recorded with the SPAD camera. It can be noticed that the signal is corrupted by noise which is due to uctuation in the intrinsic Poisson statistic of the light intensity.
Figure 4.4: Pipeline of the new algorithm.
Considering the band between 0.4 Hz and 2 Hz the noise almost disappear, as its components are at higher frequency, as shown in the lower panel of gure 4.5.
Finally, by performing FFT, a maximum was found around 1.1 Hz that means 66 bpm, that is the average heart rate in resting condition, as it can be seen in gure 4.6 .
By using this setup it was possible to calculate heart rate, but the system is still subjected to illumination power source noise, Poisson statistics and movement of the subject. These sources of noise make it di cult to calculate the HRV (heart rate variability). It was decided to develop algorithms to reduce noise and detect points of interest in the signal in order to identify all the single heart beats.
4.3 Wavelength selection
During the tests to identify the nal setup, dierent optical lters were used in order to nd out which wavelength resulted in containing the highest information related to pulse wave. Filters from 400 nm, blue light, up to 850nm, infra-red light, 50 nm steps, was used in the comparison, since this range well matches the spectral range of the SPAD camera. Each lter has 40 nm width. Five subjects were recorded using all lters. In this setup each subject cardiac activity was also monitored using a portable ECG recorder (Faros 180, see section 3.3). Records were taken always in resting conditions, and each acquisition lasted 10 minutes. To obtain a wide spectrum in the light source, dierent kind of illumination were tested and nally an incandescent lamp powered by DC source was chosen, due to its wavelengths components. The emission spectrum of an incandescent light is showed in gure 4.7
In order to determine which wavelength gave the best results, 4 parameters were calculated. The rst parameter was the accuracy in the single beat detection, based on the shape of the average beat; the second was the error in heart rate determination; the third one was
Figure 4.5: Upper panel: raw signal extracted by forehead pixels; lower panel: bandpass ltered signal.
the error in the estimation of the tachogram and the last one was the estimation of the LF/HF components, explained in section 2.2.
As previously mentioned, in this setup the Faros was used to record the ECG of the subject. This device work at a frequency of 250 Hz. Simultaneously the SPAD camera recorded a video at the frequency of exactly 416,25025 Hz. These non-multiple frequencies created a phase shift between the two devices that results in errors in synchronization. For this reason a software was created to directly control the exposure time and the integration time of the SPAD camera, in order to precisely set the acquisition frequency of the camera. It was decided to set the acquisition frequency at 100 Hz. This operation had also a smoothing eect on the signal, thus resulting in improvements on the raw signal.
Figure 4.6: Power spectrum of the extracted and ltered signal.
4.3.1 Single beat detection
The capability to detect a single heart beat is the rst parameter used to discriminate the best wavelength in terms of signal quality. Matlab will be used to develop an algorithm ( described in section 4.3.1.2) to detect each heart beat in the signal extracted by the images from the SPAD camera.
In order to achieve this result, both ECG and ltered SPAD signal are necessary. The scheme 4.8 represents the backbone of the algorithm.
4.3.1.1 Signals preparation
The signal extracted by the camera was rstly ltered with a Butterworth bandpass lter with bandwidth between 0.4 Hz and 4 Hz, and then averaged in windows of 5 samples, thus the resulting signal had an eective sampling frequency of 20 Hz.
The two signals were manually synchronized during the acquisition by hitting the Faros device at the beginning of the recordings and the stroke was visible in the accelerometer data, as explained in section 3.3. Both the signals were then resampled in order to have the same number of samples and the same time references. The results of all these processing is shown in gure 4.9. Once the two signals were overlapping a segmentation of the pulse signal was necessary.
Figure 4.7: Emission spectrum of the used incandescent lightbulb powered with DC source.
4.3.1.2 Segmentation of the signal
The following section describes the algorithm developed in order to identify each heart beat in the signal extracted by the SPAD camera.
For each signal, after applying the preprocessing step described in section 4.3.1.1, the rst 30 seconds and the last 30 seconds were removed. This operation was necessary in order to remove movement artifacts at the beginning and at the end of the acquisition, and consider only the steady state of the recorded subject. The aim of this algorithm is to normalize and overlay all the beats in the rPPG signal in order to obtain the shape of the mean beat and compute the standard deviation between all detected beats.
To segment the pulse wave it was decided to shift the wave so that between two QRS complexes of the ECG track, superimposed to the rPPG signal, a complete beat was present. In order to do so, the following procedure was implemented: a window between 120 and 180 seconds in both ECG and pulse wave signals was considered. Inside this window the sample position of all the QRS complexes in the ECG track were determined using the Pan-Tompkins algorithm [19] while in the pulse wave, the position of the maxima was determined considering the MATLAB function ndpeaks ( gure 4.10a). Subsequently, the dierence between the positions of all the detected QRS complex and the nearest maxima in the pulse wave was computed and averaged, in order to nd the mean distance between the beats in the two waves. This average allow to translate the pulse wave and obtain a better synchronization of the two signals, and also
Figure 4.8: Single beat detection algorithm. Blue line: SPAD signal processing; Red line:ECG signal processing; Green line: elaboration using both signals.
Figure 4.9: Piece of synchronized SPAD signal and ECG.
it allowed to consider the distance between two consecutive QRS complex, that depends on the heart rate ( gure 4.10b). Once the signals were perfectly matched a shift of half average beat was applied ( gure 4.10c), the whole pulse signal and ECG track was considered and it was segmented in pieces corresponding to the time interval between two consecutive QRS complex. Looking ate the gure 4.10, rPPG signal was segmented between two black peaks. These operations were necessary in order to obtain a characteristic shape of each beat of the pulse wave. Acquisition made with wavelength that did not allow to reconstruct a particular shape of the beat and which standard deviation from the median beat were higher than a threshold, were considered uninformative, as will be explained in section 5.2.1. All these operations are described in gure 4.8 and were performed for all the recorded subjects and all the optical lters. In gure 4.11 the extracted heart beat pulse wave, in two acquisitions, are plotted (the blue lines represent the mean pulse wave shape). The upper panel shows an optimal result of the beat shape, while the lower panel shows a non-recognizable shape. 4.3.2 Average heart rate
The second value considered in the determination of the wavelength with the highest infor-mation on pulse wave, is the computed heart rate. To achieve this result, after the bandpass ltering of the SPAD signal and the high pass ltering, over 0.4 Hz, of the ECG track needed to remove slow trends of the signal due mainly to movement artifacts, a FFT was performed. Results will be explained in section 5.2.1. The ideal result would show a perfect matching between the heart rate calculated from ECG and the one calculated from the pulse wave. An example of result is illustrated in gure 4.12. As it can be seen in the gure, the maximum
(a) First step
(b) Second step: sinchronization. Green line: synchronized signal.
(c) Third step: time shift. Black line: shifted signal.
Figure 4.10: Segmentation algorithm example. Blue: rPPG signal. Red, green, black: QRS time positions in the three steps.
Figure 4.11: Gray: all detected beats in the pulse wave. Blue: average pulse wave beat. Red: standard deviation of all the beats in the pulse wave signal. Upper panel shows an example of optimal beat shape, while lower panel shows discarded shape.
frequency component of both pulse wave and ECG track is the same and in particular is at 0.95 Hz, resulting in 57 beats per minute (bpm), physiological value for a resting subject. The accuracy in the determination of heart rate is calculated as the absolute dierence in the number of beats per minute.
Figure 4.12: Red: power spectral density of ECG signal; Blue: power spectral density of pulse wave extracted from SPAD camera.
4.3.3 Tachogram calculation
The third gure of merit considered in the determination of the most informative wavelength is the error in the calculation of the tachogram. For this section of the algorithm the original sampling frequency of the rPPG signal, 100 Hz, was considered (i.e. average over ve-samples window was not computed), in order to have an improved time resolution.
As explained in section 2.2, heart rate variability is important for many reasons: the mean value of the tachogram is the heart rate, its frequency components contains informations on respiration rate and, above all, information on the health of the autonomic nervous system. This kind of information is commonly extracted from an ECG, due to the typical shape of the QRS complex that make it easy to detect with high accuracy each heart beat and its temporal position. The aim of the presented algorithm was to calculate a tachogram starting from a pulse wave, mechanical phenomenon, that resulted as similar as possible to the tachogram calculated from an ECG, electrical phenomenon. Moreover it must be taken into account that
easy task, but the wave presented situation where the beat is a lot smoothed resulting in a small peak that was hard to detect. The maxima were detected using the Matlab function ndpeaks, setting two threshold, one for the temporal distance between maxima, and one for the height of the maxima.
The dierence of temporal positions of the detected maxima represented the RR-intervals and these values were saved in an array. Once the maxima had been found, the average of RR intervals was calculated. This operation gave information about the average distance between two consecutive maxima. The average was used to adjust the temporal threshold and perform a second round of searching the maxima on all the pulse wave with the constrain that there must be a maximum inside a window as wide as the calculated RR average. Figure 4.13a shows the performance of the rst round of maxima detection. In this case a beat is missed, and the gure 4.13b shows the eects of the second round, detecting the missed beat and correcting the tachogram. As a result of this rst part of the algorithm a raw tachogram was calculated. A diagram of this part of the implemented algorithm is shown in gure 4.14. 4.3.3.2 Improved tachogram estimation
After applying the steps described above, the obtained tachogram presented two main situa-tion of error: in one case it could occur that a beat was missed and the tachogram presented a large peak with respect to the baseline; in the other, it could also happen a small peak was misled and considered as a maximum, resulting in a peak under the baseline, and the algo-rithm searched for the following maximum after the previously calculated average RR interval skipping the real maximum, resulting in a high peak over the baseline. Examples of these situations are showed in gure 4.15, in particular gure 4.15a shows the former situation, a non-detected beat resulting in a high peak in the tachogram, while gure 4.15b shows the latter situation, where two consecutive peaks were misled returning an unexpected minimum
(a) First round of maxima detection. An example of missed beat is showed.
(b) Second round of maxima detection. The missed beat is correctly detected.
followed by a large maximum.
For both situations an average between 4 consecutive RR intervals was calculated (see formula 4.1).
RRaverage =
RR(i − 5) + RR(i − 4) + RR(i − 3) + RR(i − 2)
4 (4.1)
If a value was higher than the previous multiplied by a threshold, it meant a beat was misread (formula 4.2).
RR(i) > RR(i − 1) × T hr (4.2)
If this value is also higher than the previously calculated average multiplied for a threshold (formula 4.3), a single beat was missed.
RR(i) > RRaverage× T hr (4.3)
In this case this element was splitted in two equal values, in order to maintain the time position of all the tachogram. Otherwise if this value was not higher than the average (formula 4.4), the second case happened, so a small RR was detected followed by an extremely high RR interval.
RR(i) < RR(i − 1) × T hr (4.4)
In this case the higher element was summed with the previous value, the results was divided by two and this became the new value of both the considered and the previous RR interval.
RRnew =
RR(i) + RR(i − 1)
2 RR(i − 1) = RRnew; RR(i) = RRnew (4.5)
Finally in order to evaluate the error between the tachogram calculated starting from the ECG and the one extracted from the video, both tachograms were resampled with a higher sampling rate. A diagram of the explained algorithm is reported in gure 4.16.
The optimal result obtained with these operations is shown in gure 4.17, where the tachogram extracted from the signal of the remote photoplethysmography is almost the same as the one calculated by the ECG track. Quality of the algorithm is based on the root mean square error calculated between the tachogram extracted by the rPPG and the ECG.
r Pn
i=1(RRrP P G,i− RRECG,i)2
(a) Blue: rPPG tachogram before algorithm application. Red: ECG tachogram. Green: rPPG tachogram after algorithm application.
(b) Blue: rPPG tachogram before algorithm application. Red: ECG tachogram. Green: rPPG tachogram after algorithm application.
Figure 4.17: Red: tachogram extracted from the ECG track; Blu: tachogram extracted from SPAD camera video.
4.3.4 Estimation of LF/HF components
In section 2.2, it was explained the importance of the sympathovagal balance, given by the ratio of the low frequency (LF), and the high frequency (HF) components of the tachogram. These components give information on the activation status of orthosympathetic and parasym-pathetic nervous system. These components changes heavily passing from a resting to a standing condition. Studies demonstrated that tachograms extracted from ECG and PPG are almost the same in resting condition, while present great dierences in standing condi-tion. These great dierences aect mainly the LF components, while the HF components present almost the same values. Keeping that in mind, the developed setup considered the subject in resting condition, thus a comparison between the spectrum of the rPPG tachogram and the ECG tachogram is achievable.
To obtain HF and LF components a simple FFT was performed on both ECG and rPPG tachogram. In order to nd the error between the two signals LF is computed as the area under the curve between 0.04 Hz and 0.15 Hz, while HF is calculated as the area under the curve between 0.15 Hz and 0.4 Hz. The error is calculated as the dierence between the LF and HF values obtained staring from ECG track and SPAD rPPG signal respectively. An example of optimal result is shown in gure 4.18, where the overlapping between the blue and red line is clearly visible.
Figure 4.18: Red: PSD calculated from ECG tachogram; Blue: PSD calculated from rPPG tachogram.
4.3.5 The choise of the wavelegth
As will be shown in section 5.2.4, after considering these 4 parameters, results showed that the optical ter 550 nm (40 nm FWHM) , gave the best performances. Accordingly, light at 550 nm will be considered in the nal setup. For eye-safety reasons, it was decided to use also light at 850 nm, that provided high quality results without disturbing the subject.
4.4 Respiration Rate
The last parameter to be estimated using the SPAD camera is the Respiration Rate.
In this setup a device that monitor respiration is needed. It was decided to design and manufacture this device, as explained in section 3.4. The respiration device provides a gold standard of the respiration rate. Through Arduino, the value of the thermistor of the device were sent to Matlab where a low pass ltering was applied. Finally, a FFT was performed on the signal thus nding a maximum corresponding to the respiration frequency.
In a resting subject, respiration rate is around 0.2 Hz, which mean 12 breath per minute. 4.4.1 Setup
In this setup it was decided to use all the devices described in section 3 to calculate everything explained. Subject's ECG was monitored by the Faros; respiration was measured with the
Figure 4.19: Red: tachogram calculated from the ECG; Blue: tachogram calculated from the SPAD camera; Green: tachogram calculated from the RGB camera
respiration device; two cameras were used: SPAD camera, with the optical lter of 550 nm, and the RGB Basler camera. This setup allowed to compare the performances of the two cameras in calculation of tachogram and respiration rate.
4.4.2 Signals preparation
Once all the measurements were concluded the dataset contained 4 signals for each subject. The signal from the SPAD camera was ltered as explained in section 4.3.1.1; the signal from the RGB camera was ltered in the same bandwidth, 0.4-4 Hz; the signal from the ECG was highpass- tered over 0.4 Hz to remove muscular artifacts and the breath track was ltered to remove electronic noises and slow temperature changes in the band 0.05 to 4 Hz that means 3 to 240 breaths per minute.
4.4.3 Tachogram calculation
The track of the two cameras and the ECG were used to calculate the tachogram, using the algorithms explained in section 4.3.3. A rst comparison could be done, between the performances of the cameras. Figure 4.19 shows an example of the 3 tachograms.
Figure 4.20: Red: PSD calculated from ECG tachogram; Blue: PSD calculated from SPAD tachogram; Green: PSD calculated from ECG tachogram; Black: PSD calculated from respiration data.
4.4.4 Respiration rate from tachogram
Finally by performing a FFT on the tachograms a peak was found exactly matching the respiration frequency. The main issue was that the respiration frequency varies a lot in a period of ten minutes, so the spectrum of all the respiration track provided dierent peaks, each one related to dierent periods of the acquisition The same was encountered in the PSD of the tachograms. Therefore, the signals were cut in windows of one minute so that the respiration rate is quite constant and a comparison was made between all the one-minute window. Results of this experiments will be explained in section 5.2.1. An example of optimal result is shown in gure 4.20.
4.5 Led illuminator
As explained in the introduction, all this work aims to monitor a subject during driving activity. In order to avoid a high illumination inside the vehicle it was decided to design and manufacture a low intensity active illuminator to be attached in front of the driver. This device is described in section 3.5. Two versions of this device were created: one using LEDs emitting at a wavelength of 550 nm, while the other emitting at 850nm, thus in the near infrared, for eye safety reasons. First test using this kind of illumination were performed in
