Development of an EEG-based emotion recognition algorithm

UNIVERSITÀ DI PISA

Scuola di Ingegneria

Corso di laurea in

I

B

TESI DI LAUREA

Development of an EEG-based emotion

recognition algorithm

Candidato:

Chiara Scaldaferri

Relatori:

Prof. Vincenzo Positano

Dr. Davide Baldo

Controrelatore:

Prof. Enzo Pasquale Scilingo

1 | P a g e

CONTENTS

2.2.1 BRAIN AND EMOTIONS ... 9

2.2.2 ROLE OF EMOTION IN EMOTIONAL BASED EEG STUDIES ... 11

3 EMPIRICAL METHOD ... 13

3.1 EXPERIMENT ... 13

3.1.1 EMOTION ELICITATION ... 13

3.1.2 PARTICIPANTS AND SETTING ... 14

3.1.3 EQUIPMENT ... 15 3.1.4 ELECTROOCULOGRAPHY (EOG) ... 18 3.1.5 PREPARATION ... 19 3.2 SIGNALPROCESSING ... 22 3.2.1 PRE-PROCESSING ... 22 3.2.2 EYE BLINKING ... 31 3.2.3 ARTIFACTS REMOVAL ... 33 4 DATA ANALISYS ... 36

4.1 FRONTALALPHAASYMMETRYINDEX ... 36

4.2 FAAINTHE“LOTTERYGAME” ... 37

4.2.1 BASELINE REMOVAL ... 39

4.3 IMPROVEMENTSOFTHEANALYSIS ... 42

4.3.1 LOOKING FOR THE BEST REFERENCE ... 42

4.3.2 BEST PAIR OF ELECTRODES TO EVALUATE FAA ... 46

4.3.3 SUBJECTS REVERSED EMOTION ... 49

4.4 CLASSIFICATION ... 50

2 | P a g e

5.1 OUTCOMESANDDISCUSSION ... 52

5.2 FUTUREWORK ... 54

TABLE OF FIGURES ... 56

3 | P a g e

1

INTRODUCTION

Emotion recognition is a field of research spread all over the world and ever growing. Nowadays there are many techniques to reach this goal based on speech, text or facial expressions. Obviously these modalities can achieve high success rates only under very controlled circumstances, otherwise the possibility of success results is very low. Also, according to psychology, this kind of emotions are just the behavioral expression of a feeling that could be consciously adapted at the situations.

This is the reason why research has changed direction towards the physiological aspects of the emotion using brain scans obtained by EEG.

So, emotion recognition from EEG signals is expected to outperform the other modalities, since brain activity has direct information about emotion, and it is also more simple to realize because doesn't depend on any external events such as someone who speaks or makes facial expression.

This research project has been conducted to recognize emotion from brain signals measured with the EEG. As mentioned before, with EEG-based emotion recognition it has become possible actually to take a look inside the subject’s head to observe their mental state during the emotion evocation.

Specifically it has been implemented an algorithm which is able to distinguish positive and negative feelings applying the Frontal Alpha Asymmetry index on the data. With the use of the algorithm is also possible associate positive values to positive emotion and negative values to negative emotion.

The work was performed at “The Neuromarketing Labs”, a German company that has laboratories equipped with advanced technologies such as EEG, fMRI to reach a complete understanding of how the brain activity is related to the behavior of a person. In order to evoke positive and negative emotions, the participants performed a virtual lottery experiment, called the “Lottery Game”. Therefore the positive and negative emotions were delivered by winning and losing money. Then, it has been evaluated the Frontal Alpha Asymmetry index on the data because, according to Davidson (1993), it makes it possible to reveal disposition to approach and withdrawal emotions.

4 | P a g e

This project has seen different working phases that could be summarized in the setting of the experiment, the acquisition and the processing of the signals and the analysis of data.

Forty healthy women were selected from the company database for the recording but only 37 brain signals were used for the analysis. The experiment consisted in choosing to press the left or right arrow key on a keyboard. Choosing the right key, would lead to winning money (from 1 to 3 Euro).

The acquisition of the EEG signal was performed via Biosemi Active 2 signal acquisition system and the analysis of the data via Matlab.

The signal processing was the next step of the analysis necessary before proceeding with further analysis because is meant to reduce the noise affecting the recorded EEG signals. This part has included:

 the resampling from 1024 Hz to 256 Hz in order to reduce memory requirements and time needed for data processing;

 the referencing;

 the removal of the mean and trends;

 the filtering using three types of filters: a high-pass and low-pass filter for removing noise and one alpha band filter to extract alpha band [8-13] Hz of signal;

 the signal segmentation to divide signal into time-locked epochs with same characteristics such us amplitude and frequency.

 the power of neuronal data with an average moving window method;

 the artifact removal;

The data analysis, instead, has included the main analysis carried out to extract information from the recorded EEG data and to improve the results realizing an emotions recognizer as accurate as possible. Therefore, it has been evaluated Frontal Alpha Asymmetry (FAA) which is an index that reveals disposition of positive and negative emotions. From the literature, we know that brain activation is inversely

5 | P a g e

correlated to the power in the alpha band [8-13] Hz and this behavior it’s called asymmetry.[21][22] In particular, higher brain activity in the right hemisphere results into lower frontal alpha power which reflects the negative/withdrawal emotions and higher activity in the left hemisphere results into lower frontal alpha power that reflects the positive/approach emotions. In other words, positive emotions are associated with positive asymmetry and negative moods are associated with negative asymmetry.

The FAA was averaged among all events of the same type (winning or losing from 1 to 3 Euro) and all subjects allowing to obtain two trends of FAA, one for winning and one for losing shown on the same graph.

An important step of the data analysis was the removal of the baseline signal (from -0.5 sec to 0 sec). This has allowed to have a result only from the response to the stimulus. Then, at the first time, the reference was set on the average of all channels with low performance. Therefore it was necessary an improvement of the analysis to find a more optimal reference for our purpose. It results in the selection of the electrode CPz as reference. This right choice was proved also because the FAA score for the winning and the losing in the “baseline interval” did not present a statistically significant difference (paired t-test: p-value = 0.43; alpha value = 0.05).

Another finding was the best pair of electrodes to compute the FAA. The first choice, also suggested by literature, was the using of F3 and F4 electrodes, and the following testing has revealed that using this electrodes, emotions can be better recognized and classified.

In order to find the best representative window presenting positive values for winning FAA trend and negative values for losing FAA trend and a strong difference between the values of these curves, it has been conducted the paired t-test as statistical analysis. It was applied on two arrays, one for the winning and one for the losing .There were 37 values in each array corresponding to the values of the 37 subjects. The arrays were obtained by the average of the two curves in some time ranges for each subject. Despite the paired t-test denotes no statistically significant difference between the winning and losing group of data in every windows selected, it has set the window in the range of 1.1 seconds and 1.6 seconds (alpha-value = 0.05, p-value = 0.13) as this window showed the larger difference between the winning and losing FAA trends. A careful examination of data revealed that the main reason for the loss of statistical significance

6 | P a g e

was the presence among the population of several subjects presenting a negative FAA value in reaction to a winning event (and vice versa for loosing). The hypothesis, supported by literature, that this “inverted” behavior would be correlated to left handers’ subjects was rejected in our population by statistical analysis. So that, we could not find an explanation to the negative FAA values showed in reaction to a winning event (and vice versa for loosing) by some subjects. Anyway, the proposed window (1.1-1.6 sec), was the one that presented the less number of subjects with reversed emotion.

Further studies on a larger population using different protocols would be required to understand if the presence of an “inverted” response could be explained by limitations in the signal analysis procedure or by the stimulation protocol.

In conclusion, a EEG signal processing procedure and the related classifier were developed able to distinguish and associate positive values with positive emotion and negative values with negative emotion. The accuracy reached with the EEG-based emotion recognition was 83.33% in the study population.

The following chapters will present a brief description on EEG system and earlier literature on emotion recognition using brain activity (chapter 2), the setting of the experiment and the signal processing of EEG signals (chapter 3), the literature on the Frontal Alpha Asymmetry and the main analysis on the data capable to improve the results (Chapter 4). The obtained results are included in the fifth chapter together with the conclusions and the possible future developments (chapter 5). For the purposes of a perspective use, the important chapters are the third and the fourth for the development methodology and analysis of issues open for further changes to the system.

7 | P a g e

2 LITERATURE

Recognizing emotion from brain activity is becoming increasingly popular as field of research. To achieve this goal, we need information about how the brain handles emotion and information on how it is possible to measure the brain activity. There are several methods to measure the brain activity. Section 2.1 describes the electroencephalography, the method which is used for our purpose. Section 2.2 will give information about literature on emotion, how emotion is handled by the human brain and an overview on earlier literature on emotion recognition using brain activity.

2.1 ELECTROENCEPHALOGRAPHY

Electroencephalography includes the recording and the interpretation of the electroencephalogram. Electroencephalogram (EEG) is the medical record of the electric signal produced by the combined action of brain cells, or precisely, the time

course of extracellular field potentials generated by their synchronous action [1].

Postsynaptic potentials, created by interconnected networks of neurons, form local bioelectrical fields that project upwards onto the scalp where their summed bioelectrical activity can be sensed by scalp electrodes. The ‘spontaneous EEG’ is recorded in the absence of external stimulus, instead EEG generated by a response to external or internal stimulus is called an ‘event-related potential’ (ERP). In a normal subject in the awake state with the scalp electrodes the EEG amplitude is about 10–100 µV.

EEG is also used in medical field to help the diagnose on identifying of different medical conditions such as epilepsy, dementia, infections, such as encephalitis (brain inflammation), coma and in the research to distinguish various stages of sleep.

Scalp EEG activity shows oscillations at a variety of frequencies, that are called rhythms: delta (0.5–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (above 30 Hz). Determinant factors for the contribution of different rhythms are the age, the behavior and the state of alertness of the subject during the experiment. Because of this, there are many difference in EEG characteristics between the subjects. Obviously the EEG pattern changes a lot if the subject is under drugs , neuro-pathological conditions or metabolic disorders.

8 | P a g e

Delta rhythm is usually recorded during the deep sleep of the subject. The amplitude of these waves is usually large, about 75–200 µV and seems to have a strong coherence all over the scalp.

Theta rhythms are more common in rodents and the frequency range is round 4–12 Hz. The waves have high amplitude and a sawtooth shape. In humans, we can observe a theta rhythm when the subject is involved in a particular cognitive or emotional state or in some pathological states related with a slowing of alpha rhythms.

Alpha rhythms are in the range about 8-12 Hz and are most dominant in both lateral posterior sites with greater amplitude in the dominant hemisphere. Alpha blocking (desynchronization) has been significantly correlated with performance in an opposite way to theta rhythm [2]. If alpha band power desynchronizes during a task, performance is enhanced. High tonic alpha band power levels are also significantly correlated with increased performance. Alpha rhythm is suppressed when there is a transition between two states: eyes closed and eyes open. Event related brain oscillations in the alpha band have typically been divided into two narrower bands; the lower and upper alpha band. Event related desynchronization (ERD) in the lower alpha band reflects attentional demands such as alertness and expectancy, whereas ERD in the upper alpha band reflects semantic processes that are related to task performance.

Beta activity increases for states of alertness and focused attention. This fact was visible in several animals and human studies.

Finally, gamma activity is related with the mind information processing as recognition of sensory stimuli.

In general, the slowest cortical rhythms are linked with a quiescent brain and the fastest to information processing.

2.2 EMOTION

Although the science has already reached important goals in every fields of research, the investigation about the precise relationship between the human brain and the emotion is still not clear. Nowadays many theories have been developed about it, but none of them has a complete scientific approval.

9 | P a g e 2.2.1 BRAINANDEMOTIONS

The first theory proposed about the relationship between brain and emotion was from James-Lange in the 19th century. The theory states that a person first perceives a stimulus or an experience in life that cause somehow physiological changes as, for example, anxiety. The knowledge of these symptoms produces emotions in the brain. About 20 years later, Cannon and Bard disagree with the James-Lange’s theory and advanced their own theory. They state that ,when a person perceive a stimulus, the nervous impulse goes straight to the thalamus and there the message is divided in two part: the first part, directed to the cortex, creates subjective experiences as fear and the second part, directed to the hypothalamus, determines symptoms.

In 1937, another scientist James Papez found the essential error in the previous theory i.e. considering that a part of the brain was intended to split the nervous impulse. Instead, he states that there is a circuit of four structures linked to each other: hypothalamus, the anterior thalamic nucleus, the cingulate gyrus and the hippocampus. Papez circuit is responsible for the central functions of emotion and for the following peripheral expression.

Figure 1: The human brain.

In 1949, Paul Maclean developed, started by Papez model, a more accurate circuit and introduced the term limbic system.

At the previous circuit was added the following structure: orbitofrontal and medialfrontal cortices, the parahippocampal gyrus and important subcortical groupings like the amygdala, the medial thalamic nucleus, the septal area, prosencephalic basal nuclei and a few brainstem formations. In this theory was affirmed that also the perception of the world, combined with body changes after a stimulus, creates emotions and this integration of

10 | P a g e

knowledge occurs in the limbic system. A description of the brain structures involved in emotion processing follows [1]:

Amygdala

The amygdalae are two groups of neurons in the human brain who together built the most important emotional brain region: the amygdala.

The amygdala is able to recognize emotions caused by some stimulus.

When a person faces some stimulus that produces emotional load, it is recognized by the amygdala.

Anterior cingulate cortex

In the anterior cingulate cortex we can distinguish two divisions: a ‘cognitive’ part and an ‘affective’ one who is used to monitor disagreement between the functional state of the organism and any new information that might have affective consequences.

Hypothalamus

The hypothalamus is the part of the brain that manages many processes in the body, such as body temperature and the release of some hormones, like dopamine.

Insular cortex

The insular cortex is connected with emotional experience and generates conscious feelings. It is responsible for the combining of sensory stimuli to create emotional context.

Prefrontal cortex

With the term prefrontal cortex is named a front part of the brain, behind the forehead and above the eyes. It seems to have a more important role in the regulation of emotion and behavior by preventing the consequences of our actions.

For our goal, it will show in the following pages the importance of the prefrontal cortex of the human brain.

11 | P a g e 2.2.2 ROLEOFEMOTIONINEMOTIONALBASEDEEGSTUDIES

According to the previous section, we can assume that the feeling of emotions come from some parts of the brain. We would like to able to measure this emotions and to categorize them. In our research, we used EEG.

EEG uses electrodes placed outside the skull and so, the received electrical signals are not only signals from one spot, but the skull spreads out the brain activity from the whole brain. Because of this, it is hard to measure the signal that comes from deep inside the brain with this method.

After multiple researches, scientists have succeed to avoid this problem and get EEG-based emotions measurements.

One of these is Kostyunina et al. [3] who have found a relationship between different emotions and different peak frequencies in the alpha band, although every band described before represent valuable and meaningful information [4].

Moreover, we know about emotion from research by EEG measurements is the difference between the left and the right hemisphere of the brain. Basically, there are two theories about:

 Right hemisphere hypothesis: this hypothesis states that the right hemisphere is involved in processing emotion (mostly negative emotions)

 Valence asymmetry hypothesis: this hypothesis states that the right hemisphere has a major activity with negative emotions and the left hemisphere with positive emotions.

How alpha band is associated with cerebral hypo activation and beta band is associated with hyper activation was investigated also by Davidson [5] which has demonstrated in a study that frontal EEG asymmetry was related to various emotions expressions [6]. According to Davidson’s influential approach/withdraw motivational model of emotion [7], activity in left frontal lobe is related to the positive emotion and motivation behavior while activity on the right frontal cortical lobe is associated with the negative emotions and withdrawal approach [8]. In the study, the EEG was recorded while a subject was watching movie clips which aroused happiness and disgust emotions.

12 | P a g e

Higher left anterior activity was observed in the film segments associated with happiness emotions. On the other hand, greater right activity was observed in relation to the sections associated with disgust emotions.

Also Sobotka et al. [9] investigated this subject to a rewarding and punishment study analyzing the EEG activity gain in the reward and punishment trials. Greater left frontal activation was seen within reward trials. .

Thanks to the multiple studies conducted in this field of research, it has been possible to understand that the brain activity is inversely related to the alpha powers, and has been related with approach or withdrawal behavior according to the activities. In particular, higher brain activity in the left lobe indicates lower alpha power and is associated with approach behavior and positive emotions. Higher activity in the right lobe indicates lower alpha power that is associated to withdrawal behavior and negative emotions [10]. Approach- withdrawal theory explains the emotional valence in the best way. According to the theory, activation in left frontal lobe is related to motivational behavior and approaching, also including positive emotions. On the other hand, activity in the opposite lobe, which is right cortex, is related to the withdrawal and negative emotions. Under the previous statement, it has been considered an index sensible to the asymmetry of the cerebral lobes called ‘frontal alpha asymmetry’. A scientific investigation [11] has shown that 60% of resting alpha asymmetry can be considered as a stable trait. The remaining 40% has a disposition to occasion fluctuations. Several studies have been conducted in the field of emotion recognition since emotion has primary role in the rational behaviors [12], that’s the reason why a lots of behavioral studies rely on emotion recognition.

13 | P a g e

3 EMPIRICAL METHOD

This chapter introduces the main parts of the empirical method, explaining the experimental design and the signal processing of EEG signals. Section 3.1 describes the experiment and the different phases to realize it. Instead, section 3.2 will give information about the signal processing to obtain from a raw EEG signal a signal usable for the following analysis.

3.1 EXPERIMENT

In order to gain experience with the measurement and analysis of EEG signals, I have done an internship at ‘The Neuromarketing Labs’, German company based in Stuttgart. This group is involved in experiments who aim to provide their clients with innovative approaches to help them make marketing decisions based on valid, reliable and objective information on how their customers perceive advertising, product design, and prices. Their laboratories are equipped with advanced technologies such as EEG, fMRI to reach a complete understanding of how the brain activity is related to the behavior of a person.

3.1.1 EMOTIONELICITATION

The aim of the work was to create an algorithm able to assess emotion through the analysis of brain activity. Several methods exist to elicit emotions. For this study, emotions were elicited by letting participants play a “Lottery Game”. In this game, they felt good or bad emotions when they win or lose money respectively. The following paragraph shows a short description of the experiment.

The Lottery Game

During this game, the participant had to choose if to press the left or right arrow key on a keyboard. Choosing the right key, would lead to winning money (from 1 to 3 Euro). Each participant started the game with 10 Euro There was a boundary to control the subject’s current money during the experiment which constraints the amount of money

14 | P a g e

could only change between 30 and -30 Euros. Additionally, the final amount of money that the subject could have was in the range of -10 to 10 Euro.

At the beginning of the experiment there was a brief description about the game. Then, ‘Left Hand’ and ‘Right Hand’ options were shown and the subject had to select one of them by pressing right or left Arrow keys. After that, a blank black screen appeared for 2 seconds and then the result displayed for 3 sec.

Figure 2: One trial of lottery game

This part was repeated 100 trials. The data collected through these experiments have been used for further analysis.

3.1.2 PARTICIPANTSANDSETTING

Forty healthy women were selected from The Neuromarketing Labs database for the recording. Actually the participants had to perform two tasks, the first one to detect emotional behavior and the other one to predict the success of the shoes. So, the Neuromarketing Labs have preferred to select women participants because they are more conscious to fashion world. There was no limitation on the age or education. The native language of all participants was German, therefore the whole experiment was in German language. All subjects had to fill an informed consent form including the details of personal information.

EEG recordings lasted for about one hour. Subject participated at both experiments during this time: forty five minutes for the ‘Shoe experiment’, and fifteen minutes for the ‘Lottery game’.

15 | P a g e

In order to get more reliable and robust results, multi-trials EEG recording was considered. The term ‘trial’ indicates the presentation of stimuli toward subjects.

Due to the weakness of the brain signals, with an amplitude between 1 to 50 microvolt, and in order to reach a ‘clear’ response to the winning and losing events, a lot of data was necessary for evaluation. Therefore, the results of each stimulus from shoes to lottery results were presented multiple times averaged during the processing. For the ‘Shoe Experiment’ 8 repeated trials were set; in ‘Lottery game’ were set 100 total events.

For this study, we used only the data from emotional behavior experiment, such us the ‘Lottery Game’.

3.1.3 EQUIPMENT

To perform our measurements, we used the Biosemi Active 2 signal acquisition system [13]. It is a 64-channel, DC amplifier, 24-bit resolution, biopotential measurement system with Active Electrodes to record the EEG, including the Electrooculography (EOG).

Biosemi Active 2 signal acquisition system is composed of six main components to follow the data recording:

Active Electrodes and Headcaps

The Active-electrode is a sensor with very low output impedance which is plugged into headcaps electrodes holder to capture the brain activity signals. The signal is send back to the AD-box through a wire. This one follows the international 10-20 system. It also contains the CMS active electrode, which is used to replace the ground electrode. Driven right leg (DRL) passive electrode drives the average potential of the subject as close as possible to the ADC reference voltage in the AD-box.

16 | P a g e

Figure 3:Active Electrodes and Haedcaps

AD-box

It forms an ultra-compact, low power galvanically isolated front-end (close to the subject) in which up to 256 sensor-signals are digitized with 24 bit resolution, measuring signals from electrodes or other sensors connected to the subject. It is the primary part of the Biosemi Active 2 signal acquisition system.

Battery box

It is the power supply of the AD-box and Active electrodes.

Figure 4: 256-channel AD box at the top with battery below.

USB2 Receiver

The Receiver converts the optical data coming from the AD-box to an USB2 output. In addition, the USB2 receiver has a trigger port with 16 independent trigger inputs and 16

17 | P a g e

independent trigger outputs. This setup keeps the complete stimulation galvanically isolated from the subject.

Figure 5:USB2 Receiver

Analog Input Box (Optional)

The standard Active 2AD-box measures signals from electrodes or other sensors connected to the subject. But in some measurements setup, one also needs to measure additional analog signals from sources that needs to be kept isolated from the subject.

Figure 6:Analog Input Box

Software

All this systems work with acquisition software based on the LabVIEW graphical programming language from National Instruments [14].This standard package handles the basic functions such as data acquisition, displaying on the screen with all usual

18 | P a g e

scaling, references, filtering options, streaming to disk in .BDF file format and network sharing.

The USB2 Receiver is connected to a PC the parallel port. An optical cable connects the USB2 Receiver and the AD-box to prevent any electrical charge from reaching the participant. The EEG cap and the earlobe electrodes are connected to the AD-box with a cable, containing one wire for each electrode.

3.1.4 ELECTROOCULOGRAPHY(EOG)

EOG is a technique for measuring resting electrical potentials of the retinal visual cells and pigmented epithelium of human eye. It has been used to register the eye movements of the subjects because they are source of artifacts which affect the brain signals. To avoid this problem, the signal is filtered before further analysis . The eye has an electrical potential which is driven from the retinal pigment epithelium that changes in response to retinal illumination. This potential is electrically positive in the front part of eye compared to the back part. A movement of the eye from left to right causes the change of the measured potential [15].

In order to register this signal we have placed a pairs of electrodes above and below the pupil of eye and also at the left and right side of the eye.

19 | P a g e 3.1.5 PREPARATION

Before the recording session, the subject was requested to turn off the mobile phone or any other electronic device that subject had with her.

The experiment was performed into the laboratory room of 'The Neuromarketing Labs' specially crafted for the recording session with mild yellow light and normal temperature. Also the company provided headcaps in different sizes and each size had a different color. To select the best headcaps for each subject, it has been measured the circumference of their heads.

The headcaps used were made up to contain maximum 64 channels and followed international 10-20 layout system EEG standard [16].

Head size

Headcap Type

Less than 56cm

Yellow headcap

From 56cm to 60

Red headcap

More than 60

Blue headcap

Figure 8: Selection Criterion

20 | P a g e

The 10–20 system is an internationally standard method to describe electrodes locations. The numbers "10" and "20" define the distances between adjacent electrodes: these distances are either 10% or 20% of the total front–back or right–left distance of the skull.

The electrodes measure electrical potentials of the brain that refer to the brain activity. Each channel has a letter and number which indicate the lobe and hemisphere location. The lobes are respectively F frontal, T temporal, C center, P parietal and O occipital. There is also an area called 'nasion' that is located between the eyes, just above the bridge of the nose and another called 'inion' that represent the lowest point of the skull. These two areas are important because at the junction of lines between Nasion/Inion and Left/Right preauricular points should be the Cz channel.

So, the Cz channel helps to locate the headcap in a right way i.e. Cz should be in the half distance, the same for the left and right preauricular points. After setting the cap and putting it on the subject, it was fixed to a belt which was wrapped around subject’s chest. This procedure helps to prevent possible moviment of the cap.

To connect electrodes and scalp was used an electric conductive substance (gel). This methodology is meant to improve the quality of the recorded signals.

The conductive gel was injected with a syringe into the 64 electrodes holders and came into contact with the scalp of the subject. After that, 64 active electrodes were plugged into the holders and connected to AD box.

Another variables measured was the skin conductance with 2 electrodes connected to subject’s left index and to mid finger and plugged into AD-box.

Instead to obtain EOG measurement, four electrodes are connected to left, right, above and below of subject’s eyes.

Before connecting the electrodes, a quick wipe with rubbing alcohol was made to the electrode areas around the subject’s eyes.

21 | P a g e

Figure 10: Equipped subject with headcap

The two parts of the experiment were executed sequentially.

Several breaks were placed between experiments trials in which the subject could drink water and start again, whenever she felt ready to do it, pressing the button 'continue'. She was not allowed to come out of the lab room during the experiment.

For the recording of the data we used Biosemi software and for the experiment presentation we used Matlab using Psychtoolbox [17].

Each recording session generated a Biosemi .BDF file, which contains the samples for each of the 64 channels (scalp electrodes) and additional information (1 channel for GSR and 4 channels for EOG). The sampling frequency is 1024 Hz and the average of all 64 channels is set as reference.

Each channel contains the voltage between its corresponding electrode and CMS. The .BDF file format is closely related to the standardized EDF (European Data Format) for recording electrophysiological data and can be converted to .MAT MATLAB file for further analysis.

22 | P a g e

3.2 SIGNAL PROCESSING

The signals measured with EEG on a human being are a valuable source of information about brain activity, although they are weak signals and partially corrupted by background noise and electrical activity due to muscles contractions. Before using them to recognize emotion, they have to be preprocessed to be able to reduce the amount of noise.

3.2.1 PRE-PROCESSING

As the EEG signals can be corrupted by noise coming from the recording environment or from electrical activity generated outside the brain (muscular activity, eye-movements, etc.). Analysis signal preprocessing is necessary before proceeding with further analysis.

The aim of this part of the work is meant to reduce the noise affecting the recorded EEG signals. The following flow-chart shows the preprocessing applied in this study:

Data Collection Resampling and Referencing Remove mean and trends Filtering - HP Filter: 2Hz - LP Filter: 22Hz

23 | P a g e

Figure 11: Pre-processing flow chart

When the EEG data were collected, each signal has been resampled from 1024 Hz to 256 Hz in order to reduce memory requirements and time needed for data processing. This has been achieved thanks to the resample function, available on Signal Processing Toolbox software. [18]

The reference electrode used in recording EEG data is usually termed the 'common' reference for the data, if all the channels use this same reference. Typical recording references in EEG recording are one mastoid (for example, TP10 in the 10-20 System), linked mastoids (usually, digitally-linked mastoids, computed post hoc), the vertex electrode (Cz), single or linked earlobes, or the nose tip. In our case, after several tests explained in the chapter 4 , we have chosen the electrode Cpz as reference.

Epoching - based on Events - 100 trials - 5 sec Compute signal power Save into .mat file

24 | P a g e

Figure 12 : Representation of cap with a map of the electrodes

Then, the sample mean of each trial was removed with a procedure defined by the equation below:

Where indicates neuronal recorded data for all trials and samples. is the index trials,

which is 100 in the lottery game. is the index of samples.

The linear trend removal of the signal was computed whit a least-squares fit of a straight line and then, has been subtracted by the signal. In order to fit the line straight, we applied the 1-degree polynomial function:

25 | P a g e

is the mean value of each channel, corresponds to each trial and is each

sample. and are the fitting parameters and is the line function for removing trends.

Filtering is usually used to extract or remove from a signal some component or feature. In this study we used three types of filters: a high-pass and low-pass filter for removing noise and one alpha band filter to extract alpha band of signal. All the filtering performed in this study was accomplished with finite impulse response filters (FIR). The rationale for using FIR filters is that such filters preserve the signal shape because they have an exactly linear phase response and a constant group delay. In fact because of their phase response is a linear function of frequency, the FIR filters have the property that all the signal frequency components have equal delay and no phase delay distortion is introduced in to the signal. This is important for the time-critical nature of this investigation. The FIR filtering operation is characterized by the difference equation 3.4 and the filter is defined by its transfer function (equation 3.5), where h(k) are the impulse response coefficients of the filter, N is the filter order, x(n) is the input signal and y(n) is the output signal.

26 | P a g e

All the filters used in this study have a constant group delay and so they introduce in the filtered signal a predictable delay which has been compensated after each filtering process.

They were designed to minimize the maximum ripple in the passbands and stopbands (equi-ripple), using the Matlab Filter design & analysis toolbox (fdatool) [19].

Figure 13: Designing a low pass filter by fdatool with specific parameters

In the table below is showed some common parameters for designing filter.

Parameter Name Parameter Value Design Method FIR (Equiripple)

27 | P a g e Filter Order Minimum Order

Density Factor 20

Sampling Frequency 256

Frequency Unit Hz

Figure 14: Designing parameters for FIR filters

The high-pass filter is used to reduce the amplitude of the signals with lower frequencies than cutoff frequency and to let the high signal frequencies remain undistorted. The high pass FIR was realized with this following parameters: pass frequency ( : 2 Hz, stop frequency ( : 0.5 HZ, passband ripple ( :0.01,

stopband attenuation ( : 0.005, order: minimum and final order: 368.

In the filter specification, regions between and are transitions regions and

frequencies lower than are rejected.

28 | P a g e

A low-pass filter is used, instead, to pass signals with a frequency lower than a certain cutoff frequency and attenuate signals with frequencies higher than the cutoff frequency. It is composed by the following parameters: pass frequency ( : 22 Hz,

stop frequency ( : 30 Hz, passband ripple ( : 0.005, stopband attenuation

( : 0.001, order: minimum and final order: 88. Frequencies greater than are rejected.

Figure 16: The Low Pass filter specification

In order to reduce the noise and obtain a signal specifically from the interested band that is the alpha band from 8 Hz to 13Hz, we have realized an alpha band filter with these parameters: first pass frequency ( :8 Hz, second pass frequency ( :13 Hz;

first stop frequency ( :7 Hz and second stop frequency ( :14 Hz, passband

29 | P a g e

attenuation ( :0.01, order: minimum and final order: 498. In this way only the

frequencies greater than and lower than are passing through this filter and

frequencies less than and greater than are rejected.

After this analysis, we have implemented a signal segmentation to divide signal into time-locked epochs with same characteristics such us amplitude and frequency.

2 sec 3 sec Time

Figure 17: Interested signal divides into epoch in lottery game experiment

The analyzed signal from the Lottery game was an interval of 5 seconds that started 2 seconds before showing the lottery results and 3 seconds after presenting the results. We have distinguished 6 event types such winning/losing 3, 2, 1 euro and therefore we have obtained 100 epochs to treat in the further analysis.

After the dividing into epochs, we have implemented the power of neuronal data with an Average moving window method. The window was defined of 0.5 second that is equivalent to 128 samples.

Where 2N+1 correspond to 128 samples+1.

The data was organized in a standardize structure for further analysis and research. Blank screen Lottery result

30 | P a g e

Variable Type Details

Filename Char Name of file

nbChan Integer Number of channels for EEG

eventChannel Integer Number of channels including external ones such as EOG and GSR

Srate Integer Sampling rate of signal

chanLocs Structure Brain region according to each channel

chLabels Cell Name of each channel by order

Reference Vector Reference channel

referenceType Char Type of reference

Events Structure Include type and latency information for each event

Date Char Date of the experiment

Epochs Structure Signal epochs

totalEvents Integer Number of recorded events

Figure 18: Data Structure

After the dividing into epochs, we have implemented the power of neuronal data with an Average moving window method. The window was defined of 0.5 second that is equivalent to 128 samples.

31 | P a g e 3.2.2 EYEBLINKING

The EEG record contains many types of artifacts such as an event or process which has not its source in an examined organ. One of this type of artifacts is eye blinking. However the amplitude of the electrooculographic signals is only six-times greater than EEG signals, there is a large interference because of short distance between sources of these signals. The eye artifact is best seen in first two channels Fp1 and Fp2. To detect this artifact we have used the signal from the vertical electrodes of EOG.

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 -150 -100 -50 0 50 100 150 200 250 Time [s] S ig n a l [V ]

Eye blinking effect

Signal F3 Signal VEOG

Figure 19: Eye blinking effect on F3 electrode before alpha filtering

As showed in the figure above, when the signal F3 and VEOG are filtered only with low pass (22 Hz) and high pass (2 Hz) filter the blinking artifact has a big effect on the signal from F3 electrode (electrode of the prefrontal cortex of brain).

However to remove this artifact we have not used any specific method because applying the alpha band-pass filter we noticed the effect is removed.

32 | P a g e -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 -150 -100 -50 0 50 100 150 200 250 Time [s] S ig n a l

Eye blinking effect

Power F3 Signal VEOG

Figure 20: Eye blinking effect on F3 electrode after alpha filtering

Also the signal from VEOG in frequency domain confirmed this assumption because, as showed the figure below, the main information of the artifact signal is not included in the range of frequencies selected by the alpha band-pass filter.

33 | P a g e -25 -20 -15 -10 -5 0 5 10 15 20 25 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 Frequency [Hz] |S ig n a l| VEOG FFT Figure 21: Signal in frequency domain from VEOG electrodes

3.2.3 ARTIFACTSREMOVAL

Artifacts are somewhat more difficult to remove because they are not present all the time, and not always in the same electrodes. Since artifacts may heavily contaminate the recordings, it is important to minimize their effect on EEG signals.

It’s possible to identify and reject artifacts containing epochs by visual inspection [7], but in this work we have realized an automated method to obtain this result.

We have evaluated for each participant the mean and the standard deviation of all trials in each sample:

34 | P a g e

is the power matrix for all samples and trials, is the index of sample, is the index

of trial, is a vector which contains the mean of trials at each sample, is a vector

contains the averaged value of standard error for all trials at each sample, is the total

number of trials and is the total number of samples.

After this we have set up the threshold for each sample as it is showed in the equation below (because of the power is a positive number, we have just used the upper boundary threshold):

is a vector contains all the threshold values at each sample.

For each trial, we have computed the percentage of samples that exceed the threshold. If the signal exceeded the threshold for more than 5% of the total samples, it was considered as a noisy trial.

The algorithm developed was composed of the following steps:

 initialize matrix and vector

 for each sample evaluating of : o the average of all trials: o the standard error of all trials:

35 | P a g e

o the threshold of all trials o for each trial,

 If , ; Otherwise

We have removed only the trials that met the following criteria:

, ; Otherwise

In which M was the number of all samples. So, as mentioned before, when signal were beyond the threshold for more than 5% of the total samples, it was removed from the data set. The significance level was chosen to balance between two situations:

 Higher accuracy could cut off too many trials and over fit the final result;

 Lower accuracy may keep the noisy trials for analysis data;

For these reason, we have used a medium accuracy level such as 95%.

36 | P a g e

4 DATA ANALISYS

This chapter presents the main analysis carried out to extract information from the recorded EEG data and to improve the results and to realize an emotions recognizer as accurate as possible. Section 4.1 will explain the knowledge about the Frontal Alpha Asymmetry index and section 4.2 the application of the index at the EEG data. Section 4.3 will give information of the improvements applied on the algorithm and section 4.4 will introduce final results through a basic classification.

We initially have scheduled 40 subjects for the experiment, but three subjects experienced technical problems during the recordings and their data cannot be analyzed. Finally, 37 subjects’ data were considered for the analysis.

4.1 FRONTAL ALPHA ASYMMETRY INDEX

After the data has been filtered and artifacts has been removed, it was necessary evaluated an index to extract features containing all the information needed to recognize emotions.

According to Davidson (1993), while right posterior regions of the brain specialize in the perception of affective stimuli of both positive and negative valence, both left and right anterior regions of the cortex may be involved in the experience of emotion. More specifically, the left frontal regions may be more active during the experience of approach-related (i.e. positive) emotions and the right frontal regions may be more active during the experience of withdrawal-related (i.e. negative) emotions.[20]

Frontal Alpha Asymmetry (FAA) is an index that reveals disposition of positive and negative emotions. From the literature, we know that brain activation is inversely correlated to the power in the alpha band [8-13] Hz and this behavior it’s called asymmetry.[21][22] In particular, higher brain activity in the right hemisphere results into lower frontal alpha power which reflects the negative/withdrawal emotions and higher activity in the left hemisphere results into lower frontal alpha power that reflects the positive/approach emotions. In other words, positive emotions are associated with positive asymmetry and negative moods are associated with negative asymmetry.

37 | P a g e

So, as mentioned before, there is a strong correlation between the Frontal Alpha Asymmetry index and emotion. [23]

Frontal Alpha Asymmetry can be obtained using two methods showed in the following expressions:

The first expression measures the frontal alpha asymmetry score as comparing of two channels’ alpha power, normalized by their sum of alpha power.[24]

The second one measures instead the difference of log alpha power.[25]

For the further analysis, it has been used the first expression to evaluate the index.

4.2 FAA IN THE “LOTTERY GAME”

As already mentioned, the goal of the study was to recognize and separate distinctly the emotions elicited by the winnings and the losings of the subjects based on the FAA reaction. We have started the work with an hypothesis: positive value of FAA score means positive emotion (winning) and negative value of FAA score indicates negative emotion (losing).

The FAA score has been obtained by the method below using two frontal channel: F3 for the left region and F4 for the right hemisphere. At the beginning this choose was only supported by literature, but, in a second time, we have run some tests, showed in the following pages, to validate this theory.

The FAA was averaged among all events of the same type (winning or losing from 1 to 3 Euro). The types which are losing and winning 3, 2, 1 Euro are also averaged and the

38 | P a g e

trial time is 5 sec. -2 in the figure 23 and 24 indicates 2 sec before presenting the stimuli, and +3 means 3 sec after stimuli presentation.

Our goal was to develop a classifier able to distinguish between positive and negative emotions. So, we were interested at the signal after the 0 second in which occurs the lottery result presentation. The highlighted portion of the signal shows the separation between winning and losing witch are respectively positive and negative values.

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 -0.04 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.04 0.05 0.06 Time(s) F AA

FAA. Subjs: 37 Ref: CPz

-3 -2 2 3

39 | P a g e -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 -0.04 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.04 Time(s) F AA

FAA. Subjs: 37 Ref: CPz

Winning (3,2) Losing (3,2)

Figure 24: Trend of FAA score averaged

4.2.1 BASELINEREMOVAL

In order to analyze the emotional reaction due to the lottery outcome, it is important to reference the signal to the brain activity which was occurring before the stimulus onset (presentation of the lottery’s result). To that end, the baseline, which is the FAA average on a window before stimulus onset, was removed from the signal. The baseline was calculated through computing the mean of the signal for each trial in the baseline interval and subtracted from whole signal. In this way the pre activity of the signal before the stimulus shows up has been deleted.

There were many intervals we considered as baseline interval for example [-2,0] , [-1.5, 0], [-1, 0] and [-0.5, 0] seconds. The best separation amongst positive and negative emotions happened when the interval [-0.5, 0] was used as baseline. Based on these results, [-0.5, 0] was selected as baseline interval to remove for further analysis.

40 | P a g e -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 -0.06 -0.05 -0.04 -0.03 -0.02 -0.01 0 0.01 0.02 Time(s) F AA

FAA. Subjs: 37 Ref: CPz. Baseline Removed

Winning Losing

Figure 25: Trend of FAA score after application of the baseline removal

Figure 23 shows the FAA reaction to winning and losing after the baseline removal. We can notice that the trends are as expected. In fact, after presenting the stimulus ( winning or losing of the money ) i.e. after second 0, we can identify a clear window in which the winning trend assumes positive value and the losing trend assumes negative ones. Also in this window, roughly between 1 and 2 seconds, the trends are opposite.

In order to find a more precise window, we decided to carry out a paired t-test. This way we could reach a better statistically significant difference.

This t-test compares one set of measurements with a second set from the same sample. Every hypothesis test requires the analyst to state a null hypothesis and an alternative hypothesis:

H0 (null hypothesis) that states the two groups are not statistical different; H1 (alternative hypothesis) that states the two group are statistical different;

The hypotheses are stated in such a way that they are mutually exclusive. That is, if one is true, the other must be false; and vice versa.

Then, assigned a significance level called alpha value (in this case 0.05), we have: 1) α < 0.05 refusal H0;

41 | P a g e

2) α ≥ 0.05 not refuse H0.

From the software for statistical analysis we obtain the p-value as result of the t-test. This value is a measure of the acceptability of the null hypothesis. In fact, if the p-value is small means that the value observed in the sample is very different from what expected by the null hypothesis, then it is unlikely that the null hypothesis is true.

We managed the data by applying a paired t-test on two arrays, one for the winning and one for the losing .There were 37 values in each array. The arrays were obtained by the average of the two curves in the time ranges below for each subject. The table shows the best results.

RANGE [sec]

H

P

Figure 26: Table of the paired t-test of winning and losing arrays in different windows

As seen in the table the results are not ideal because they do not show the statistical difference between these two group, i.e. H=0.

In order to choose an “event window” for the analysis, it has been used the one from 1.1 sec to 1.6 sec in a tested set of ranges much larger than 0.4 sec because on a quality level presented the largest difference between the curves.

It is recommended that in order to reach an ideal outcome we should experiment the same test on more subjects and decrease the number of subjects with reversed emotion, explained in the follow paragraphs.

42 | P a g e

4.3 IMPROVEMENTS OF THE ANALYSIS

In the following paragraphs, we present the work done to achieve better classification results. The goal is to find the EEG processing algorithm which leads to a clear FAA post-stimulus response (i.e. which allows differentiate the most between positive and negative emotions).

4.3.1 LOOKINGFORTHEBESTREFERENCE

In order to extract out of the FAA the most information regarding emotional reaction, a preliminary analysis has been conducted to detect which reference could be the best for this work. There is no 'best' common reference site. Some researchers claim that non-scalp references (earlobes, nose) introduce more noise than a non-scalp channel reference though this has not been proven to our knowledge. If the data have been recorded with a given reference, they can usually be re-referenced to any other reference channel or channel combination.[27]

So, the first step was the searching in literature what was done before, to have an idea about the fundamental characteristics of a reference.

So, the parameters considered when looking for an optimal result were:

 The trends of the FAA for Winning and Losing as expected by hypothesis

 Statistical results (paired t-test)

 Number of electrodes as small as possible (easier to implement)

Our goal is to determine which processing would lead to the highest separation among winning and losing FAA trends. Also, we would like to achieve such result keeping the number of the electrodes as small as possible to reduce the difficulty of the implementation and also that the difference between the two groups would be significant statistically. To prove this, we have run the paired t-test to groups of the 37 subject considering the two conditions of the subject i.e. the winning and the losing for each sample. So, for each sample we obtained winning and losing arrays of 37 values, one value from the FAA trend per subject. If the hypothesis H0 was accepted the groups were not statistically different, otherwise they were. In a statistical analysis, we keep

43 | P a g e

considering also the p-value of the t-test, in fact, given a null hypothesis H0, this can be accepted or rejected based on the value of p-value.

A strict definition of p-value is presented below:

“The p-value is the level of significance assigned, i.e. a measure of evidence against the null hypothesis”.

Then, assigned a threshold value, for us 0.05, we have: 1) α < 0.05 refusal H0;

2) α ≥ 0.05 not refuse H0.

As smaller is the p-value, as bigger is the evidence against the null hypothesis. [28]

At first, we tested the average reference guided by literature but in a second time, we have proved some others possibilities based either on literature then on considerations about how would be possible to get a more neat signal:

 Cz reference

 Cpz reference

 Pz reference

 Average among AFz, Fz, FCz, Cz, CPz, Pz, POz reference

 Average among AF3, AFz, AF4, FC3, FCz, FC4, C3, Cz, C4, CP3, CPz, CP4 reference

 Average among C5, C3, C1, Cz, C2, C4, C6 reference

44 | P a g e

Figure 27: on the left: Average among AFz, Fz, FCz, Cz, CPz, Pz, POz reference; at right:

Average among AF3, AFz, AF4, FC3, FCz, FC4, C3, Cz, C4, CP3, CPz, CP4 reference

Figure 28: Average among C5, C3, C1, Cz, C2, C4, C6 reference

The average of all the channels was refused as approach later because it didn’t reveal the best results by the data and implied a severe time to be implemented.

Instead, the best results reached were obtained with the Cpz and Pz references. At the end, it has been chosen the Cpz reference because the obtained winning and losing curves had a more common trend when the subjects are not under stimulus than the other one. This effect is very important for the analysis because reflects the original idea in which the result visible is only a consequence of the stimulus of the winning and losing of the subject.

45 | P a g e

Figure 29: Trend of FAA score with Average reference (the color fuchsia

indicates the samples have statistically difference)

Figure 30: Trend of FAA score with Pz reference (the color fuchsia

46 | P a g e Figure 31: Trend of FFA score with Cpz reference (the color fuchsia

indicates the samples have statistically difference)

As seen in the graph the t-test shows a radical difference before the event, which is illustrated in fuchsia. This value is considered to be as a casual and random result which to us is not reliable, and moreover it’s occurred in a short range of time, less than 0.4 seconds.

4.3.2 BESTPAIROFELECTRODESTOEVALUATEFAA

After a first result of the application of our method on the EEG signal, we have started to implement algorithms trying to improve the previous results.

One of these was based on the idea that selecting a different pairs of electrodes to evaluate the Frontal alpha asymmetry index could lead to find a bigger difference in the window indicated of the winning and losing trends.

So, according to the literature and our suggestions based to find two channels having the same distance to the reference to obtain a symmetric effect, it has been chosen some pairs of electrodes, listed here: F3/F4, F1/F2, F7/F8, FC5/FC6, C5/C6 and P5/P6.

47 | P a g e

For each one we have applied the paired t-test between winning and losing array obtaining H and P value.

To run the t-test, we have considered the average of the winning and the losing curves in the window between 1.1 and 1.6 sec because was the window selected before.

So, for each pair of electrodes, we have evaluated the FAA for each subject and we have obtained 37 values for winning and for losing doing the average of the signal in the window selected.

The table shows the t-test results for each pair of electrodes with alpha value 0.05.

RANGE [sec]

H

P

C5-C6

F3-F4

F7-F8

FC5-FC6

F1-F2

P5-P6

Figure 32: Table of t-test results of the pairs of electrodes selected for FAA score

The table shows as the best result the C5/C6 electrodes pair, the only one that reach the statistical significance. However, the significance on the C5-C6 pair is due to an almost constant difference between signals, not significantly modulated by the stimuli. In particular, a significant difference was detected also in the baseline period where no stimulation was provided to the subject.

For these reasons, we have been chosen the F3/F4 electrodes pair, that is the pair suggested by literature. In fact, also in this situation, the winning and losing curves with F3/F4 electrodes had a more common trend, when the subjects are not under stimulus, than the other choice. To justify the graphical suggestion, it has been applied a t-test between the winning and losing trends in the window of the baseline period that showed not statistically significant difference between the two curves in that period using F3/F4 electrodes.

48 | P a g e

As already done, the paired t-test was executed exactly like before but the window chanced from “event window” [1.1 1.6] sec to “the baseline window” [-0.5 0].

The paired t-test has showed a p-value = 0.4333 with an alpha value of 0.05, then a not statistically significant difference between the two groups.

The same test on the data from C5/C6 electrodes showed a statistically significant difference indicated by the color fuchsia in the figure 33.

49 | P a g e Figure 34: Trend of FAA score using F3/F4 electrodes

4.3.3 SUBJECTS REVERSED EMOTION

During the analysis of the data, we would like to improve the outcome to understand the response of the subjects one by one to find out the number of the subjects that had a small difference between the two curves i.e. winning and losing. In order to do this, we have implemented an algorithm that evaluated, at the first time, the average of the FFA score in the window from 1 second to 1.6 second separately for the winning and the losing curves and then applied a difference respectively between these two values. From this algorithm, we have obtained a figure in which there are unexpected negative values. This result means that some subjects response in a opposite manner to a winning and a losing of money i.e. positive values are related to a losing and negative values are related to a winning.

50 | P a g e 33 127 27 181 54 120 162 145 1 28 80 46 156 142 163 59 55 164 165 159 172 170 180 173 167 175 138 144 13 174 70 128 123 171 124 84 168 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 SubjCode F A A W in -L o se

FAA. Subjs: 37 Ref: CPz DeBased StartWin: 1.1 EndWin: 1.6

Figure 35: Value of difference between the winning and losing FAA score in the window from 1.1 to 1.6 seconds

In the figure above, the numbers in the horizontal axis represent the way in which they were memorized in the company database.

The first hypothesis to explain this fact was found in literature, in fact there are many articles that talk about emotion reversed in left-handers’ brain.[26]

In order to approve this theory, it has been done a meticulous research in the company database to find if this assumption was the explanation of the phenomenon.

As none all of the subjects presenting the occurrence demonstrate to be left handed, so the hypothesis was refused.

During this study, the cause of this unexpected result, has been deeply investigated but we could not find a definitive answer.

4.4 CLASSIFICATION

As final analysis, it has been proved the results obtained with a classifier to figure out how much of the previous hypothesis of the work had been achieved.

51 | P a g e

In the classifier showed in the figure below we can recognize the average values of emotions caused by winning and losing of 3,2 and 1 euro on the part of the subjects in the window from 1.1 sec to 1.6 sec. To obtain the average values in this figure we averaged the FAA trends of winning and losing for all the subjects. And then again averaged them in the window between 1.1 and 1.6 sec. The threshold used was zero because in the hypothesis a positive emotion (winning) has to be related with a positive value and negative emotion (losing) has to be related with a negative value. The result has indicated the recognition of the emotion has failed only for the winning of 1 euro letting us to reach an accuracy of 83.33%. This evidence found justification in the very small amount of money won by the subjects in this case that doesn’t arouse a strong emotion.

As expected the graph shows clearly a radical difference in the window selected between the positive and negative emotions due to winning and losing respectively. And this is the outcome that we were trying to prove.