BRAIN SCANS AS A TOOL TO PREDICT SUCCESS OF FASHION ITEMS

UNIVERSITÀ DI PISA

SCUOLA DI INGEGNERIA

Corso di Laurea in Ingegneria Biomedica

TESI DI LAUREA MAGISTRALE

Brain Scans as a Tool to Predict Success of

Fashion Items

Relatori: Candidato:

Prof. Vincenzo Positano Federica Milicia Dott. Davide Baldo

Controrelatore:

Prof. Enzo Pasquale Scilingo

Contents

ABSTRACT _________________________________________________________ 1 CHAPTER 1 - INTRODUCTION __________________________________________ 5 1.1 NEUROMARKETING AND DECISION MAKING ______________________________ 5 1.2 ELECTROENCEPHALOGRAPHY __________________________________________ 6 1.3 FRONTAL ALPHA ASYMMETRY _________________________________________ 9 1.4 GOAL OF STUDY ____________________________________________________ 11 CHAPTER 2 - MATERIALS AND METHODS ________________________________ 12

2.1 SUBJECTS AND SHOES COLLECTION ____________________________________ 12 2.2 EQUIPMENT AND ENVIRONMENT SETTING ______________________________ 13 2.3 PROCEDURE _______________________________________________________ 16

2.3.1 EXPERIMENTAL DESIGN – THE SHOES EXPERIMENT __________________________ 19 2.3.2 SHOES EXPERIMENT BASE MODEL ________________________________________ 20

CHAPTER 3 - DATA ANALYSIS _________________________________________ 21 3.1 EEG PRE-PROCESSING _______________________________________________ 21 3.2 EEG ANALYSIS _____________________________________________________ 30

3.2.1 FAA EVALUATION ____________________________________________________ 31 3.2.2 DATA STRUCTURE IN MATLAB __________________________________________ 35

3.3 FEATURES EXTRACTION _____________________________________________ 36

3.3.1 SHOE RESPONSE ANALYSIS _____________________________________________ 37 3.3.2 PRICE RESPONSE ANALYSIS _____________________________________________ 42

3.4 CLASSIFICATION ____________________________________________________ 45

3.4.1 SUPPORT VECTOR MACHINE ____________________________________________ 46 3.4.2 LINEAR DISCRIMINANT ANALYSIS ________________________________________ 49 3.4.3 LINEAR CLASSIFIER ____________________________________________________ 51 3.4.4 NEAREST-NEIGHBOR CLASSIFIER _________________________________________ 52

3.6 REDUCING NUMBER OF TRIALS________________________________________ 61 CHAPTER 4 - RESULTS _______________________________________________ 68 CHAPTER 5 - CONCLUSIONS___________________________________________ 70 5.1 RELATION BETWEEN SHOES EXPERIMENT AND LOTTERY GAME _____________ 71 List of Figures ______________________________________________________ 73 List of Tables ______________________________________________________ 75 Bibliography_______________________________________________________ 76

1

ABSTRACT

For thousands of years human beings have been trading goods and services. Trying to understand what happens in the mind of a customer has been akin to understanding a “black box” via trial and error. Very recent advances in neuroscience indicate a paradigm shift in this age-old challenge. The rising fields of neuroeconomics and neuromarketing shed light on decision processes and preferences of the consumer mind. Based on the analysis of the brain activity, this pilot study utilizes neuroscientific EEG recordings to understand and predict customers’ preference of shoes and relates this neuroscientific measure to sales in the market.

The analysis was based on the analysis of EEG Frontal Alpha Asymmetry (FAA). This is a well known EEG parameter, which has previously been used as an indicator of the emotional state, presumably being able to differentiate between approach and avoidance motivation. FAA score was derived based on the difference of alpha power between left and right hemispheric brain activity.

An experiment, called ‘Shoes Experiment’, was run to predict success of shoes. Forty women took part in the experiment and thirty shoe models for women were investigated. The shoes were already sold in the market and knowing the sales data allowed to categorize each shoe according to its performance. The 30 shoe models were categorized into two different groups, 15 of them were highly successful and 15 were not successful.

For brain activity recording was utilized a Biosemi Active 2 signal acquisition system with 64-channel, DC amplifier, 24-bit resolution, biopotential measurement system with Active Electrodes to record the EEG and the Electrooculography (EOG). The Shoes Experiment protocol included several parts: first of all, the participants read the informed consent and signed it; then, the 30 shoes were shown in real to the subjects and we asked them to fill a list (a questionnaire) which specifies ranking for each shoe (from 1 to 5); after this, the subjects, separately, were brought to the recording room and the EEG recording could started.

The EEG recording part was divided in 8 sessions in which all 30 shoes were shown randomly. Each session was called trial and the showing of one shoe was

2

called epoch. During a single epoch, the subject fixed a small white cross in a black screen before the onset of the epoch, saw a shoe for 3 seconds without any information about the selling price, saw the shoe with its price for the following 3 seconds, and then was asked to explicitly state whether or not they would buy the shoe by selecting ‘yes’ or ‘no’ for 1 sec.

The EEG data was collected for each subject and analysed offline using Matlab. The data analysis procedure was divided in pre-processing and analysis stage.

The pre-processing stage was conducted before further analysis. The purpose of this step was to reduce the noise, thus the signal that EEG registers but which is not due to cerebral activity. This signal is called artifact and can be divided into physiological artifact or extra-physiological artifact. The EEG data from 64 electrodes were down sampled from 1024 Hz to 256 Hz; the electrode CPz was chosen as reference for all the other electrodes; the average of the signal and the linear trend were removed; the data was filtered with a low-pass filter (22Hz the cutoff frequency), high-pass filter (2Hz the cutoff frequency) and band pass filter to extract the alpha waves EEG frequencies (from 8Hz to 13Hz); EEG data was segmented in epochs and the further analysis were computed on the signal from 2 seconds before the shoe was presented to 6 seconds after; the signal power was computed for each epoch using a moving average filter; to remove epochs contaminated by muscles activity and non-cerebral activities, an automatic method was implemented based on the individuation of a threshold.

To use the EEG signal as an indicator of decision making, it was necessary to extract an index from the signal. In this work, the Frontal Alpha Asymmetry (FAA) was chosen. The index that was computed uses the asymmetry of the brain activity by evaluating the difference of alpha power between left and right hemispheric brain activity. The FAA was computed using midfrontal sites: F3 to measure the activity of the left-frontal hemisphere and F4 for the right one. In order to investigate how FAA changes after the stimulus is presented on screen, a baseline was computed for each epoch and subtracted to the FAA. The baseline represents the FAA averaged in a 500 milliseconds window just before each shoes appears on screen. To reduce the intra-subject variability and noise, all epochs for each shoes were averaged together and

3

from the 30 FAA obtained, knowing which shoes were good and which bad (from the sales data), the average of the only good shoes and of the only bad shoes was computed.

The features chosen in this work were identified within the single epoch. Because a single epoch is divided in 2 temporal periods within the shoes are shown twice, we searched two features for the different periods, the shoe response and the price

response. The shoe response is the FAA trend between 0 and 3 sec; the price

response is the FAA trend between 3 and 6 sec. On these features were computed other analysis: to select a window in both features, the classification rate, the correlation coefficient, the t-test and the standard deviation were computed. The best windows turned out to be the 0.5s-1.5s for the shoe response feature and 0.2s-0.6s for the price response feature. The accuracy was 73.33% for both features.

After the definition of the optimal window for EEG signal, the Two Fold Cross-Validation method was used to assess the overall accuracy of the classification algorithm. The method defined training data set, for which the class was known a priori, and a defined validation data set, for which the class was also known a priori and which is used to determine the accuracy of the prediction. The method consists in applying the classification learned with the training data to the validation data and then to evaluate the classification accuracy rate using a supervised learning algorithms computed on training data. To divide the data in the two groups, the Bootstrap method was chosen. For each combination of the training and validation data sets, the classification algorithm was applied and the resulting accuracies were averaged to evaluate the global accuracy of the proposed methods. Four classifiers were implemented and tested: the Support Vector Machine, the Linear Discriminant Analysis, the Linear Classifier and the Nearest-Neighbor Classifier. In this work, two features were available, the FAA response to the presentation of the shoe and the FAA response to the display of the prices. Thus, it was possible to define both a 1-dimensional classification (using the two features separately) and a 2-dimensional classification (using both features). One of the classifiers is mono-dimensional in nature and was applied to the data as a 1-dimensional classifier (Nearest-Neighbor Classifier). The others were applied as a 2-dimensional classifiers (SVM, LDA, Linear Classifier).

4

Also, in the multi-dimensional classifiers the self-report data (the questionnaire data) was used as a feature to build a 3-dimension classifier where the other 2 features are the FAA responses to the view of the shoe and the presentation of its price.

Among proposed classifiers, the best accuracy of 72.87% was obtained using a 1-dimensional classifier (Nearest-Neighbor Classifier on the shoe response feature). For comparison, the traditional marketing method based on the use of a questionnaire correctly classify only 16 shoes.

The experiments were repeated reducing the number of trials and acceptable results were obtain with 7 or 6 trials but not better than the result with 8 trials.

In conclusion, the EEG based assessment of customers’ preference of shoes outperformed the standard approach based on customer interviews by questionnaire. The best classifier resulted the one-dimensional Nearest-Neighbor Classifier applied to the shoe response feature.

5

CHAPTER 1 - INTRODUCTION

1.1 NEUROMARKETING AND DECISION MAKING

It is well known that people’s subjective reports are both incomplete and include after-the-fact rationalizations for decisions that miss or misinterpret what was really driving their behavior. They frequently lack direct conscious access to what determines their feelings, intentions and actions. Many contextual and motivational factors influencing behavior may operate at an implicit emotional level, yet may directly impact perceived attractiveness, product preferences and brand loyalties. In light of this, the interest of the industries has become to understand the consumer perceptions through a deep understanding of their emotions. They want to understand the emotional impact of products on customers, the decision making and, the final deed, the buying impulses (M.Smith, G.Calvert, R.Wright, & al., 2013).

The decision making are made in a non-random way, even if with a certain irrationality. More precisely, the buyer decision processes are cognitive processes and are function of the marketing action taken by companies. This means that although it is never possible ‘to see’ a decision, it is possible to act on it. To achieve this goal, a new field of marketing research has sky-rocketed in interest during the past ten years, called Neuromarketing. The question about how consumers are making their choices seems to be one of the core topics in this new field.

Neuromarketing is the combination of cognitive neuroscience and economics which reveal directly how decision making takes place and how the brain is influenced by marketing stimuli. Its goal is to creep into the mind of the consumer, to decipher his behavior, his tastes and especially his desires, because they are the key to de success. Neuromarketing researches and strategies are able to solve marketing and advertising challenges.

Then, why using neuroscience to understand consumers? Traditional methods are using surveys, interviews, focus groups, where people are overtly and consciously reporting on their experiences and thoughts. On the other side, the unconscious side of consumer behavior is largely unmeasured in traditional methods. Neuroscience,

6

and Neuromarketing as well, has the potential to understand the unconscious drivers of choices (Figure 1). In recent years, several neuroscience studies demonstrated that brains scans are better predictors of customer behaviour than self-reports (Berns & Moore, 2012).

Figure 1: Traditional methods against Neuroscience methods

The techniques that Neuromarketing uses to measure costumer behavior include eye-tracking, biometrics, voice analysis, facial expression coding or facial electromyography (EMG), various forms of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI).

To date, there isn´t a better technique than the other, the use of one technique over another depends on the object under examination and on the goal of the research.

1.2 ELECTROENCEPHALOGRAPHY

The Electroencephalogram (EEG) is used to measure the bioelectrical activity projected onto the scalp via volume conduction from cortical as well as subcortical sources within the brain. Postsynaptic potentials created by interconnected networks of neurons create local bioelectrical fields that project upwards onto the scalp where their summed bioelectrical activity can be sensed by scalp electrodes.

7

A typical EEG signal has an amplitude between 10 µV and 100 µV when measured from the scalp (Malmivuo & Plonsey, 1995). EEG is commonly used as a clinical tool in the diagnosis of cerebral dysfunction (tumors, epilepsy, localize origin of seizures), legal determination of coma and brain death (cerebral silence), and to distinguish various stages of sleep (Sherwood, 2009). In Brain Computer Interfaces (BCI) features extracted from the EEG are used together with pattern recognition and classification methods in order to create a direct interface between the brain and computers (Friman, Volosyak, & Grasesr, 2007). Nowadays EEG is associated with economics field for marketing purposes which emotion recognition or attention as a result of advertising as well as prediction of purchase decision.

Scalp EEG activity shows oscillations at a variety of frequencies. Several of these oscillations have characteristic frequency ranges, spatial distributions and are associated with different states of brain functioning. Most rhythmic activity within the EEG can be divided into five classical bands (Figure 2): delta, theta, alpha, beta and gamma (Andreassi, 2006).

Delta rhythm (0-4 Hz) is high in amplitude (20-200 µV) and is usually recorded in

frontal sites for adults and posterior sites for children during slow wave sleep (Andreassi, 2006).

Theta rhythm (4-8 Hz, 20-100 µV) is usually widespread over the entire cortex but

has been found to have an important contribution over the prefrontal cortex in various cognitive tasks (Andreassi, 2006).

Alpha rhythm (8-12 Hz, 20-60 µV), or Berger rhythm, is the EEG base rhythm and is

recorded in an awake patient with closed eyes among parietal and occipital electrodes; if the patient opens the eyes the alpha activity disappears and it is replaced by an activity of low voltage called beta rhythm (Andreassi, 2006).

Beta rhythm (15-30 Hz) is common during mental or physical activity and has an

amplitude of 2-20 µV; it is associated with physiological arousal and response to threat (Andreassi, 2006).

Gamma rhythm (30-70 Hz) is widespread over the entire cortex; it is often observed

over the sensorimotor cortex as a mechanism for integrating sensory and motor processes during programming of movement (Andreassi, 2006).

8

Figure 2: Four typical dominant brain normal rhythms, from high to low frequencies

In conventional scalp EEG, the recording is obtained by placing electrodes on the scalp with a conductive gel or paste, usually after preparing the scalp area by light abrasion to reduce impedance due to dead skin cells. Many systems typically use electrodes, each of which is attached to an individual wire. Some systems use caps or nets into which electrodes are embedded. Electrode locations and names are specified by the International 10–20 system for most clinical and research applications. Each electrode is connected to one input of a differential amplifier (one amplifier per pair of electrodes); a common system reference electrode is connected to the other input of each differential amplifier. These amplifiers amplify the voltage between the active electrode and the reference. To date, the digital EEG machines are the most widespread and the amplified signal is digitized via an analog-to-digital converter, after being passed through an anti-aliasing filter. Analog-to-digital sampling typically occurs at 256–512 Hz in clinical scalp EEG; sampling rates of up to 20 kHz are used in some research applications.

9

Despite the relative poor spatial resolution, EEG possesses multiple advantages like a very high temporal resolution, hardware costs lower than the other techniques (fMRI, SPECT, PET, etc.), it is silent and does not aggravate claustrophobia.

1.3 FRONTAL ALPHA ASYMMETRY

The Frontal Alpha Asymmetry (FAA) was identified by Davidson as a correlate of approach/avoidance, a high-level emotional dimension that plays an important role in many aspects of human psychology. The left- and right-anterior brain regions (Figure 3) are part of two separate neural systems underlying approach (e.g. joy, happy, interest) and withdrawal (e.g. fear, anger, disgust) emotions, respectively (Davidson, 1990).

Figure 3: (A) The four lobes of the human cerebral cortex. (B) Top view of human cerebral cortex; the brain is largely divided into left and right hemispheres

Decrease in synchronous activity in the alpha band in adults is associated with activation, thus when populations of cortical neurons become activated there is a decrease in amplitude. Relatively greater left frontal activity, either as a trait or a

10

state, indicates a propensity to approach or engage a stimulus, while relatively greater right frontal activity indicates a propensity to withdraw or disengage from a stimulus. From this, a parameter used to distinguish approach and withdrawal emotions is computed as the difference between the right frontal EEG alpha power and the left frontal EEG alpha power. Being the EEG power inversely related to cortical activation, negative EEG asymmetry scores reflect greater relative right frontal EEG activity and, therefore, negative emotions.

A consumer´s purchase decision involves a tradeoff between the pleasure derived from consumption and the pain of paying. That is, paying money triggers a perception of loss, even though it has also been suggested that money spent in buying goods is not coded as a loss (Bateman, Kahneman, & al., 2005). In regard to motivational tendencies, anticipatory pleasure of acquisition should be associated with approach motivation, whereas anticipatory pain of paying should be associated with withdrawal motivation. A situation where approach motivation elicited by a preferred product exceeds withdrawal motivation should be associated with an affirmative purchase decision.

The FAA analysis became, in the last decade, one of the most used methods for neuromarketing (Briesemeister, Tamm, & al., 2013).

Some neuromarketing research papers examined how approach motivation as indexed by electroencephalographic (EEG) asymmetry over the prefrontal cortex predicts purchase decision when brand and price are varied (Ravaja, Somervuori, & Salminen, 2012). In a within-subjects design, the participants were presented purchase decision trials with 14 different grocery products (seven private label and seven national brand products) whose prices were increased and decreased while their EEG activity was recorded. The results showed that relatively greater left frontal activation (i.e., higher approach motivation) during the pre-decision period predicted an affirmative purchase decision.

11

1.4 GOAL OF STUDY

The aim of this study was to develop a tool able to predict sales of shoes before they are launched in the marketplace using EEG signals. The challenge was to understand which shoe models are more attractive for the costumers with an accuracy higher than the traditional marketing methods and look at whether it was possible to identify specific neural responses that would predict interests and subsequent choices. With this predictor, the company could produce only the profitable shoes, obtaining a production line optimization, increasing profit and being sure of satisfy his clients.

12

CHAPTER 2 - MATERIALS AND METHODS

2.1 SUBJECTS AND SHOES COLLECTION

Forty healthy participants took part in the study after providing written informed consent. Since the shoe models were for women, all the participants were females with no limitation on the education or age (average = 29.3 years old, SD = 11.5 years old).

In the field of neuroscience the research sample size is much smaller than that in other fields. Because of the average sample size of published neuroscience research is 25 – 40, the number of samples was chosen to obtain a statistically accurate analysis when using brain data.

Thirty shoe models for women were investigated in this experiment.

Each pair of shoes was sold in stores of a shoe retail chain in the central European market, at full price from August to November 2013. The sales data were obtained for each pair of shoes.

A parameter “success” was used to categorize each shoe according to its performance. This parameter was defined as:

𝑆𝑢𝑐𝑐𝑒𝑠𝑠 [%] =

𝑠ℎ𝑜𝑒𝑠 𝑝𝑟𝑜𝑑𝑢𝑐𝑒𝑑

𝑋 100

According to sales figures (Figure 4), the 30 shoe models were categorized into two different groups, 15 of them were highly successful (“Success” average 77 %) and 15 were not successful (“Success” average 19 %).

13

Figure 4: The parameter “Success” for the two groups of 15 shoes. Error bars indicate standard error of the mean

The shoes selection was decided by the company and, in order to have uniform distribution for the shoes, we asked to have the same number in each group (15 “successful” shoes and 15 “unsuccessful” shoes).

In this work, the “successful” shoes were called “good shoes” and the “unsuccessful” were called “bad shoes”.

2.2 EQUIPMENT AND ENVIRONMENT SETTING

For brain activity recording was utilized a Biosemi Active 2 signal acquisition system with 64-channel, DC amplifier, 24-bit resolution, biopotential measurement system with Active Electrodes to record the EEG and the Electrooculography (EOG). The system consists of 6 main components to pursue the process of data recording: Active Electrodes and Headcap

By integrating the first amplifier stage with a sintered Ag-AgCl electrode, extremely low-noise measurements free of interference are now possible without any skin

14

preparation. The Active Electrode is a sensor with very low output impedance without any capacitive coupling between the cable and sources of interference. The electrode is connected to the headcap electrodes holder (Figure 5) to capture the brain activity signal, then the signal is sent to the AD-box (explained later). The cap contains the Common Mode Sense (CMS) active electrode and the Driven Right Leg (DRL) passive electrode, which are used to replace the ground electrodes used in conventional systems. These 2 electrodes form a feedback loop, which drives the average potential of the subject (the Common Mode voltage) as close as possible to the ADC reference voltage in the AD-box. With BioSemi systems, every electrode or combination of electrodes can be the "reference"; the choice is made entirely in software. When no reference is selected in software, the signals are displayed with respect to the CMS electrode (BioSemi).

Figure 5: Headcap for 64 electrodes

AD-box

It forms an ultra-compact, low power galvanically isolated front-end (close to the subject) in which up to 256 sensor-signals can be digitized with 24 bit resolution,

15

measuring signals from electrodes or other sensors connected to the subject. It is the primary part of the Biosemi Active 2 signal acquisition system (Figure 6) (BioSemi).

Figure 6: Biosemi 256-channels AD-box

USB2 Receiver

The Receiver converts the optical data coming from the AD-box to an USB2 output. In addition, the USB2 receiver has a trigger port with 16 independent trigger inputs and 16 independent trigger outputs. This setup keeps the complete stimulation galvanically isolated from the subject and allows easy setup of EP/ERP measurements, and event logging (BioSemi).

Analog Input Box (Optional)

The standard Active 2 AD-box measures signals from electrodes or other sensors connected to the subject. But in some measurements setup, one also needs to measure additional analog signals from sources that needs to be kept isolated from the subject (BioSemi).

Battery box

16 Software

All this systems are working with acquisition software based on the LabVIEW graphical programming language from National Instruments. This standard package handles the basic functions such as data acquisition, displaying on the screen with all usual scaling, references, filtering options, streaming to disk in .BDF file format and network sharing (BioSemi).

Environment Setting

It was shown that emotional response to video/images stimuli is affected by environmental parameters such as room lighting (Knez, 1995) and the temperature (Anderson, Deuser, & Neve, 1995). We set these parameters to be as natural and comfortable as possible to limit possible sources of stress.

The experiment was carried out in a lab room limiting as much as possible the noise from outside.

2.3 PROCEDURE

Participants ran the experiment individually. First of all, they read the informed consent, they signed it and they were given instructions about the experiment. This was divided in 3 parts:

1. Showing to the participants the 30 shoes in real: the subjects were taken to a mock shoe-shop where 30 pairs of female shoes were displayed. The shoe arrangement was randomized each time to avoid biases due to location or order. Each subject was free to walk around to touch and feel the shoes as if they were in a real shop. Each shoe was accompanied by a price. Participants could take as long as they needed to rate the shoes. This was done to maintain the real-life context and be able to avoid anchoring effects;

2. Questionnaire (Behavioral Experiment): asking the participants to fill a list which specifies ranking for each shoe; the ranking was from 1 to 5 (1 = didn’t like the shoe at all, 5 = liked the shoe very much and would like to

17

buy) (Figure 7). This part of the experiment was intended to simulate the engaging environment in actual shoe shops and to determine whether or not we could develop a system able to provide a better prediction of sales data; 3. Doing the EEG recording: once the subjects had rated all the shoes, the EEG

experiment could started. The data was collected for each subject and analysed offline.

Figure 7: Questionnaire in German language. Schuhnummer is the serial code of the shoes; Wertung is the score; Schlecht is bad; Gut is good

For the recording session, the subject was conducted to the lab room (Figure 8). The head circumference was measured and the suitable headcap was selected. The headcaps are in different sizes and it is important choose the correct one to obtain a good EEG recording. The headcap used was with 64 channels that follows the international 10-20 system (Figure 9). The 10–20 system (or International 10–20 system) is an internationally recognized method to describe and apply the location of scalp electrodes in the context of an EEG test or experiment. This method was developed to ensure standardized reproducibility so that a subject's studies could be compared over time and subjects could be compared to each other. This system is based on the relationship between the location of an electrode and the underlying area of cerebral cortex. The "10" and "20" refer to the fact that the actual distances

18

between adjacent electrodes are either 10% or 20% of the total front–back or right– left distance of the skull.

Figure 8: EEG acquisition during the experiment. The subject is in the lab room and wears the headcap; he is in front of the screen where the shoes and the prices are shown

Each site of the cap has a letter to identify the lobe and a number to identify the hemisphere location. The letters F, T, C, P and O stand for frontal, temporal, central, parietal, and occipital lobes, respectively. Two anatomical landmarks are used for the essential positioning of the EEG electrodes: first, the nasion which is the distinctly depressed area between the eyes, just above the bridge of the nose; second, the inion, which is the lowest point of the skull from the back of the head and is normally indicated by a prominent bump.

19

Figure 9: Electrodes locations based on 10-20 system

The duration of the experiment was of roughly 45 minutes, in which EEG signals were recorded and, in addition, 4 EOG electrodes and GSR were recorded.

Before this time, the subject was guided through a brief description about the experiment on the screen.

The subject had to work only with left and right arrow key on the keyboard to choose whether to buy or not to buy the shoe on the display.

During the experiment the subject had many little break within they could drink, moving and blinking.

2.3.1 EXPERIMENTAL DESIGN – THE SHOES EXPERIMENT

This part of the experiment was divided in 8 sessions in which all 30 shoes were shown randomly. Each session was called trial and the showing of one shoe was called epoch.

During a single epoch, the subject fixed a small white cross in a black screen before the onset of the epoch, saw a shoe for 3 seconds without any information about the selling price, saw the shoe with its price for the following 3 seconds, and then was asked to explicitly state whether or not they would buy the shoe by selecting ‘yes’ or

20

‘no’ for 1 sec (Figure 10). The 30 shoes were displayed in one experiment for 8 times (or trials) then the number of epochs was 240.

Figure 10:Experiment Design for each shoe in a single epoch

2.3.2 SHOES EXPERIMENT BASE MODEL

According to Bateman’s results (Bateman, Kahneman, & al., 2005), where approach emotions during the pre-decision period predicted an affirmative purchase decision, another experiment called Lottery Game, developed at The Neuromarketing Labs, was used as pattern for the shoes experiment.

The Lottery Game demonstrated that a money win elicits approach emotions while a money lost elicits withdrawn emotions and then it was possible to identify a temporal window in which distinguish the difference of the two emotions using the FAA index.

The hypothesis of the shoes experiment was that the view on the screen of a shoes liked by the subject would determine a FAA reaction similar to the one evoked by winning at a lottery and viceversa for a disliked shoe model.

Therefore, the shoes experiment was designed in the same way of the Lottery Game experiment to be comparable with this latter.

The data analysis procedure was developed following the structure of the base experiment: Pre-processing and Analysis stage.

21

CHAPTER 3 - DATA ANALYSIS

All the data were analyzed using Matlab.

Instead of 40 subjects, 37 subjects were analyzed because 3 people were excluded for technical problems during the recording of EEG data.

The data analysis procedure, according to the Lottery Game experiment, was divided in:

 EEG Pre-processing

 EEG Analysis

3.1 EEG PRE-PROCESSING

The following pre-processing (Figure 11) was conducted before further analysis. The purpose of this step was to reduce the noise, thus the signal that EEG registers but which is not due to cerebral activity. This signal is called artifact and can be divided into physiological artifact (e.g. ocular, muscle) or extra-physiological artifact (e.g. 50/60Hz interference, DC component).

22

After the recordings, EEG data was down sampled to 256 Hz from an initial frequency of 1024 Hz to reduce the computational time and disk space.

Then, the referencing step was implemented. Ideally, the electrode measurement should only represent the electrical activity on a precise spot. However, since voltage is a relative measure, the measurement is compared to the voltage at a reference site. Unfortunately, this results in measurements that reflect the local activity, but also the activity at the reference site (Figure 12). Because of this, the reference should be chosen such that the activity at the reference site is almost zero. The nose, mastoids and earlobes are typically used (Hagemann, Naumann, & Thayer, 2001). Working with Biosemi it is necessary choose another reference site (instead of CMS selected automatically) to obtain the full CMRR, useful to reduce the noise. In this work, the CPz electrode was adopted as reference to follow the theory of the Lottery Game experiment.

Figure 12: EEG signals as a combination of brain activity, reference activity and noise

In order to remove any trend from the signal, the average of the signal recorded at each electrode, computed over the all recording time, was subtracted to the signal and the linear trend was removed using a 1 degree polynomial function (Equation 3.1), subtracting it from the signal (Figure 13).

23

𝐹(𝑖) = 𝑎

𝑌

(𝑖) + 𝑎

𝑌

(𝑖) =

𝑁

The i index represents the channel, the j index represents the sample, 𝑎₁ and 𝑎₂ are the fitting parameters and 𝐹 is the line polynomial function computed for each signal coming from the channels to be subtracted.

Figure 13: EEG signals before the subtraction of the trend (up) and after the subtraction (down)

Because many artifacts have frequencies higher than 22 Hz and lower than 2 Hz, an high pass filter with cut off frequency in 2 Hz and a low pass filter with cut off frequency in 22 Hz were used. The filters were implemented with the Matlab Toolbox (Figure 14) and were designed as FIR filters which are inherently stable and can easily be designed to have a linear phase. The knowledge of the delay introduced from the filter allows to shift the filtered signal to get the result without delay. The filters were designed to minimize the maximum ripple in the passband and stopband (Figure 15 & Table 1).

24

Figure 14: FDA Tool for the low pass filter

25

Filter Fstop Fpass Dstop Dpass

Low pass 30 Hz 22 Hz 0.001 0.005

High pass 0.5 Hz 2 Hz 0.005 0.01

Table 1: Parameters of the LP and HP filters; D is the ripple in passband and in stopband

A band pass filter was used to extract the alpha waves EEG frequencies; the range was among 8 and 13 Hz (Figure 16 & Table 2).

Figure 16: BP Filter parameters specification in Matlab FDATool

Filter Fstop1 Fpass1 Fpass2 Fstop2 Dstop1 Dpass Dstop2

BandPass 7 Hz 8 Hz 13 Hz 14 Hz 0.01 0.01 0.01

26

EEG data was recorded during all the experiment but, since we were interested in the ongoing brain activity while the subject was solving the task, the pre-processing was computing on the signal from 2 seconds before the shoe was presented to 6 seconds after. The selection of the samples indicative of the task was possible using the Biosemi trigger system. Such, EEG data were segmented into 8 sec long epochs (Figure 17).

Figure 17: Epoch divided in 3 windows, the concentration period, the shoe response window and the price response window

The pre-stimulus window (baseline) was defined in the range between -2 and 0 sec and the post-stimulus window between 0 and 6 sec, where the time 0 was the moment when the stimulus was presented.

The signal power was computed for each epoch using a moving average filter:

𝑦(𝑖) =

2𝑁−1

(𝑥(𝑖 + 𝑁) + 𝑥(𝑖 + 𝑁 − 1) + ⋯ + 𝑥(𝑖 − 𝑁))

where 𝑥 is the square of the EEG signal ([µV2]), 𝑦 is the power of the EEG signal filtered ([µV2]) and 2𝑁 − 1 is the width of the moving window. The width of the moving window was defined as 0.5 seconds which is equivalent of 128 samples.

One of greatest nuisances are those artifacts resulting from oculomotor activity (Figure 18) since this signal can be orders of magnitude larger then brain-generated electrical potentials (Joyce, Gorodnitsky, & al., 2004). These artifacts are almost inevitable because subjects cannot well control spontaneous eye movements or blinks.

27

Further, the instruction to inhibit eye movements or blinks may seriously distort brain activity.

Figure 18: Comparison of the vertical Elettrooculogram signal and the one frontal electrode signal

Several methods have been developed to cope with the problem of ocular artifacts. The most popular approaches are the correction of ocular artifacts by means of regression analysis (Lins, Picton, & al., 1993) and the independent component analysis (Jung, Humphries, & al., 1998).

Evaluating the Fast Fourier Transform of the vertical EOG was clear that the significant frequency range of this signal came up till 5-8 Hz (Figure 19). Therefore, the alpha waves filter (8-13 Hz) completely removed eye-blinks artifacts (Figure 20).

28

Figure 19: Fast Fourier Transform of the vertical Electrooculogram

29

Another relevant artifact is usually caused by muscles activity and non-cerebral activities (the highest trends in Figure 21). It is common to remove the noisy epochs manually with visual inspection but, in this work, an automatic method was implemented based on the individuation of a threshold.

Figure 21: 240 Epochs of the Power Signal F3 electrode of one subject

Two of the 64 electrodes were used for the algorithm, the power of F3 and F4 electrodes. A threshold was evaluated for each electrode, according to the Equation 3.4:

𝑇ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑(𝑒, 𝑖) = 𝐴𝑉𝐺(𝑒, 𝑖) + 𝑐𝑜𝑒𝑓𝑓 × 𝑆𝑇𝐷(𝑒, 𝑖) Equation 3. 4

where e is the electrode and i is the time instant in the epoch. The average and the standard deviation were computed for each sample. The coefficient used was equal to 4. A result of this automatic artifact rejection algorithm is shown in Figure 22:

30

Figure 22: 240 Epochs of the Power Signal F3 electrode with threshold

The epochs were rejected if exceeded the threshold a number of times dependents from a sensitivity parameter. The rejection of an epoch for one electrode prejudiced the rejection from the other electrode. The mean rejection rate was 10%.

3.2 EEG ANALYSIS

To use the EEG signal as an indicator of decision making, it was necessary to extract an index from the signal. In this work, the Frontal Alpha Asymmetry (FAA) was chosen as time domain analysis.

FAA had been used in various studies related to psychophysiology such as to detect the depression (Henriques & Davidson, 1991), fear (Kostyunina & Kulikov, 1996), stress (Sulaiman, Taib, & al., 2010), happiness (Davidson, 1990) and so forth. The findings are based on the frontal cortical activations. High right frontal activity has been reported to be found during depression while high left frontal activity was detected during happy emotional states.

31

The index that was computed uses the asymmetry of the brain activity by evaluating the natural log of alpha right power minus the natural log of alpha left power (Equation 3.5):

𝐹𝐴𝐴 = 𝑙𝑛(𝑃𝑜𝑤𝑒𝑟_𝑟𝑖𝑔ℎ𝑡) − 𝑙𝑛(𝑃𝑜𝑤𝑒𝑟_𝑙𝑒𝑓𝑡) Equation 3. 5

Another expression of asymmetry can be used by evaluating a ratio (Equation 3.6), expression used in this work:

𝐹𝐴𝐴 = (𝑃𝑜𝑤𝑒𝑟_𝑟𝑖𝑔ℎ𝑡 − 𝑃𝑜𝑤𝑒𝑟_𝑙𝑒𝑓𝑡)/(𝑃𝑜𝑤𝑒𝑟_𝑟𝑖𝑔ℎ𝑡 + 𝑃𝑜𝑤𝑒𝑟_𝑙𝑒𝑓𝑡) Equation 3. 6

The possible scores can be greater than zero (reflects a left hemisphere activity), lower than zero (indicates a right hemisphere activity). The middle point equivalent to zero is the symmetrical activity situation where both hemispheres equally activate.

3.2.1 FAA EVALUATION

As in previous research (Davidson, 1990) and based on the outcome of the Lottery Game experiment, in this work the FAA was computed using midfrontal sites: F3 to measure the activity of the left-frontal hemisphere and F4 for the right one (Figure 23).

The reference choice fell on CPz, according with the Lottery Game experiment, because it is at the same distance from F3 and F4.

32

Figure 23: Position of F3, F4 and CPz channels on the headcap

33

The FAA in Figure 24 was obtained using the signal power in the Equation 3.6. Thus, this index perform the emotion elicited by the shoe the subject sees in that epoch.

In order to investigate how FAA changes after the stimulus is presented on screen, a baseline was computed for each epoch and subtracted to the FAA. The baseline represents the FAA averaged in a 500 milliseconds window just before each shoes appears on screen (Figure 17). This procedure helps reducing the effect of brain activity before stimulus onset on the FAA changes due to stimulus presentation. To reduce the intra-subject variability and noise, all epochs for each shoes were averaged together (Figure 25).

Figure 25: FAA mediated on 37 subjects for each shoe

From the 30 FAA obtained, knowing which shoes were good and which bad (from the sales data), the average of the only good shoes and of the only bad shoes was computed (Figure 26).

34

Figure 26: FAA mediated on 37 subjects for the ´good´ shoes and the ´bad´ shoes

Figure 26 shows exactly the difference of the FAA values for the good and the bad shoes during the epoch time of 8 seconds. To test the validity of the result a statistical approach was used, the paired-sample t-test (Figure 27):

 H0 = not significant difference, the two trends belong to the same population

 H1 = significant difference, the two trends belong to different populations The t-test was computed sample by sample, after to have verified the normal distribution of the data.

35

Figure 27: FAA mediated on 37 subjects for the ´good´ shoes and the ´bad´ shoes with the statistically significant difference

The pink points symbolize that is not possible to accept the null hypothesis, thus the two trends are statistically different with a p-value lower than the significance level (paired-sample t-test, p-value = 0.0372, alpha-value = 0.05).

The following paragraphs are going to delve into the data analysis part that was done to individuate the specific window within the epoch time that distinguishes the good from the bad emotions.

3.2.2 DATA STRUCTURE IN MATLAB

In order to keep the loading and the use of all the information produced as easy as possible was created a structure array, FAAaggregated.

This array contains the following fields:

 SubjsCode: subject number

 ArtifPerc: artifact epochs percentage

 data: FAA for each subject clustered for the shoe type

36

 dataDeBase: FAA for each subject clustered for the shoe type after the rejection of the baseline

 reference: reference index on Biosemi

 referenceType: reference label

 date: creation of the structure

 srate: sampling frequency

 time: epoch time

 ArtifDetParam: parameters used for the artifact detect algorithm

 baselineStart: baseline start time

 baselineEnd: besaline final time

 EEG: information about the EEG data (no. electrodes, reference, no. events expected, etc.)

 GrandAvg: Data structure mediated on the subjects

 GrandAvgDeBase: dataDeBase structure mediated on the subjects

3.3 FEATURES EXTRACTION

Technically, a feature represents a distinguishing property, a recognizable measurement, and a functional component obtained from a section of a pattern. Extracted features are meant to minimize the loss of important information embedded in the signal. In addition, they also simplify the amount of resources needed to describe a huge set of data accurately. This is necessary to minimize the complexity of implementation, to reduce the cost of information processing, and to cancel the potential need to compress the information (Al-Fahoum & Al-Fraihat, 2014).

The features chosen in this work were identified within the single epoch. Because a single epoch is divided in 2 temporal periods within the shoes are shown twice, we searched two features for the different periods, the shoe response and the price

37

The shoe response is the FAA trend between 0 and 3 sec; the price response is the FAA trend between 3 and 6 sec (Figure 28).

Figure 28: FAA values for the signals without the baseline signal (dataDeBase)

Figure 28 depicts the average FAA for good (in black) and bad (in red) shoes. At time zero shoes are presented and at time 3 seconds the prices is also displayed together with the shoe. Regarding the time window [0 3] seconds, where only the shoe was on screen, it is possible to see that there is a statistically significant difference (pink points on the figure) between the two trends. These few seconds can be used to identify in a significant way the group identity of a shoe.

3.3.1 SHOE RESPONSE ANALYSIS

The first analysis was computed on the shoe response feature (which is the FAA signal in the window [0 3]s during the shoe presentation).

Starting from the FAAaggregated structure array, data and dataDeBase were aggregated to obtain two 3-dimensional matrices of 30x2049x37, called DataMatrix and DataMatrixDeBase (Figure 29 (a)). The choice of the window consisted in

38

selecting the samples corresponding to the period of time desired and doing the average on those samples, obtaining the Emotions_AllSubjectsAllShoes matrix 30x37 (Figure 29 (b)). Then, this matrix was divided in two arrays, corresponding at good and bad shoes and on each array was evaluated the average on the shoes, obtaining the Emotions_Good and Emotions_Bad arrays of 1x37 (Figure 29 (c, d)). From Emotions_AllSubjectsAllShoes was also obtained the Emotions array 30x1 (Figure 29 (e)) that represent the FAA value of each shoe. All these matrix/array were obtained from the normal data and the data without baseline (data and dataDeBase).

Figure 29: Representation of the DataMatrix or DataMatrixDeBase matrix (a), Emotions_AllSubjectsAllShoes matrix (b), Emotions_Good and Emotions_Bad arrays(c, d), Emotions array (e)

To select a window in the feature, the width of this was chosen and many widths were tested. For each width, the window was shifted within [0 3]s and different parameters were examined to identify the optimal window (Table 3):

 Classification rate: the percentage of shoes correctly classified (sales data are known);

 Correlation coefficient: between the feature and the sales data;

 T-test result: p-value of the statistic test;

39

To compute the statistical parameters, different data representation were used, Emotions, Emotions_Good and Emotions_Bad (Figure 29).

Classification

The one-dimensional supervised classification could be implemented because sales data of the shoes were known. Each shoe was identified with a FAA value and with a label (good or bad shoe) (Figure 29 (e)). The FAA value was obtained choosing a temporal window within the shoe response and doing the average of these samples. The threshold used for the classification was the average of the array. The classification rate, or accuracy, was computed according to Equation 3.7:

𝐴𝐶𝐶 = ((#𝑐𝑜𝑟𝑟𝐺𝑜𝑜𝑑𝑆ℎ𝑜𝑒𝑠 + #𝑐𝑜𝑟𝑟𝐵𝑎𝑑𝑆ℎ𝑜𝑒𝑠))/(#𝑡𝑜𝑡𝑆ℎ𝑜𝑒𝑠) 𝑋 100 Equation 3. 7

where #𝑐𝑜𝑟𝑟𝐺𝑜𝑜𝑑𝑆ℎ𝑜𝑒𝑠 is the number of good shoes the algorithm classifies correctly, #𝑐𝑜𝑟𝑟𝐵𝑎𝑑𝑆ℎ𝑜𝑒𝑠 is the number of bad shoes classified correctly and #𝑡𝑜𝑡𝑆ℎ𝑜𝑒𝑠 is the total of shoes in the experiment.

Correlation

In statistics, the Pearson´s Coefficient is a measure of the linear correlation (dependence) between two variables X and Y, giving a value between +1 and −1 inclusive, where 1 is total positive correlation, 0 is no correlation, and −1 is total negative correlation.

According to the hypothesis that good shoes (with high sales data) would elicit approach emotions and then high FAA values, the correlation between the emotion elicited by viewing a shoe and the sales data should be positive.

T-test

After making sure that data were normally distributed, the paired t-test was computed. Paired samplest-test typically consists of a sample of matched pairs of similarunits, or one group of units that has been tested twice (a “repeated measures”t-test).

In order to determine whether or not the emotions (FAA) elicited by the good shoes were statistically different from the one elicited by the bad shoes, a t-test was run.

40

An optimal outcome, according to the hypothesis of the experiment, was to obtain a statistically significant difference between the two groups of data with a p-value lower than the significance level (0.05).

The test was run between FAA of good and bad shoes (Figure 29 (c,d)). Standard Deviation

In statistics and probability theory, the standard deviation measures the amount of variation or dispersion from the average. A low standard deviation indicates that the data points tend to have values concentrated around the mean; a high standard deviation indicates that the data points are spread out over a large range of values. The standard deviation was computed on the FAA value for each subject averaged on the good shoes (Figure 29 (c)) and the FAA value for each subject averaged on the bad shoes (Figure 29 (d)); the two SD were summed. The outcome has to be the smallest possible to be an optimal value.

Width Window Window Start Window End

ACC Corr P-value Corr P-value t-test Stand. Dev. 0.4 s 0.5 s 0.9 s 60% 0.2673 0.0767 0.0426 0.0787 0.6 s 1 s 63.33% 0.2345 0.1061 0.0579 0.0795 0.75 s 1.15 s 70% 0.2001 0.1446 0.1185 0.0794 0.5 s 0.45 s 0.95 s 60% 0.2576 0.0847 0.0509 0.0772 0.55 s 1.05 s 63.33% 0.2380 0.1026 0.0593 0.0783 0.7 s 1.2 s 70% 0.2137 0.1284 0.1077 0.0782 0.6 s 0.55 s 1.15 s 66.67% 0.2366 0.1040 0.0684 0.0773 0.65 s 1.25 s 70% 0.2289 0.1118 0.0969 0.0770 0.7 s 0.5 s 1.2 s 66.67% 0.2451 0.0958 0.0698 0.0761 0.55 s 1.25 s 70% 0.2442 0.0967 0.0768 0.0761

41 0.8 s 0.45 s 1.25 s 70% 0.2512 0.0903 0.0762 0.0749 0.55 s 1.35 s 73.33% 0.2490 0.0922 0.0922 0.0748 0.9 s 0.45 s 1.35 s 73.33% 0.2553 0.0867 0.0901 0.0739 1 s 0.5 s 1.5 s 73.33% 0.2724 0.0727 0.0938 0.0730 1.2 s 0.4 s 1.6 s 70% 0.2769 0.0693 0.1073 0.0713

Table 3: Statistical parameters for different windows from data without baseline (shoe response feature)

The best window turned out to be the 0.5s-1.5s (Table 3) from data without baseline (dataDeBase) because of the good compromise among the parameters.

Figure 30: One-dimensional classification for the best window

The Figure 30 shows that, with the average of the FAA values (Figure 29 (e)) as threshold, more than the 73% of the shoes were correctly classified.

42

This type of classification, actually, is not very correct because using the total of the data doesn’t define a reliable classifier. The 3.4 paragraph explains how this problem was solved using the Two Fold Cross-Validation method.

However, the period of time from 0.5 to 1.5 sec, identified from the statistical analysis, can be taken as the temporal window in which the “good” and the “bad” signals are separated the most (Figure 31).

Figure 31: Optimal window during the shoe response, 0.5:1.5 s

3.3.2 PRICE RESPONSE ANALYSIS

The Figure 32 shows the price response (which is the FAA response to viewing the shoe together with its price) for the good and the bad shoes. As depicted in the figure, there is a small period of time ([0.2 0.8]s) in which the signals have an opposite trend. This effect is not clear why appends. It is possible that the idea to buy and, therefore, to spend money for the shoe we like tends to elicit negative emotions as we would need to spend money, a resource, to get the shoe model we like and desire (Bateman, Kahneman, & al., 2005). What is clear is that the averages of the two

43

groups change at the same time and it is still possible to recognize an optimal window to distinguish the two signals.

Figure 32: Price response from 3s to 6s

The emotional reaction (FAA) to the viewing of the shoe with its price was analyzed in the same way of the previous feature.

From the 3-dimensional matrices of 30x2049x37, called DataMatrix and

DataMatrixDeBase, were selected the samples corresponding to the time period of

the price response, from 3 to 6 sec (Figure 28).

The manipulation of the matrices was the same as for the shoe response feature (Figure 29) and was performed for the signals before and after the baseline removal. The statistical analysis provided another table (Table 4) for this feature in which was possible to individuate the window for the price response as a good compromise between the statistical parameters.

For this feature an opposite correlation between the values of the FAA and the sales data was found. In the Figure 32 it is possible to see an inverse trend (window [0.2

44

0.8]s) of the emotional reactions (FAA) compared to our expectations. In order to better compare the statistical parameters of this feature with the previous parameters (Table 3), the absolute values of correlation and the complementary values of accuracy (100-accuracy%) were taken.

Width Window Window Start Window End

ACC Corr P-value Corr P-value t-test Stand. Dev. 0.4 s 0.2 s 0.6 s 73.33% 0.2469 0.9058 0.0833 0.0711 0.25 s 0.65 s 66.67% 0.2683 0.9241 0.0617 0.0723 0.3 s 0.7 s 63.33% 0.2705 0.9258 0.0587 0.0733 0.5 s 0.15 s 0.65 s 73.33% 0.2296 0.8889 0.1070 0.0687 0.2 s 0.7 s 66.67% 0.2447 0.9038 0.0903 0.0670 0.25 s 0.75 s 63.33% 0.2428 0.9019 0.0952 0.0711 0.6 s 0.1 s 0.7 s 73.33% 0.2095 0.8667 0.1396 0.0664 0.15 s 0.75 s 66.67% 0.2153 0.8734 0.1348 0.0675 0.7 s 0.1 s 0.8 s 66.67% 0.1834 0.8340 0.1961 0.0653 0.15 s 0.85 s 63.33% 0.1768 0.8250 0.2144 0.0661 0.8 s 0.1 s 0.9 s 63.33% 0.1464 0.7770 0.2826 0.0640 0.9 s 0.1 s 1 s 60% 0.1154 0.7282 0.3532 0.0626

Table 4: Statistical parameters for different windows from data without baseline (price response feature)

The optimal window individuated was the one from 0.2 to 0.6 sec (Figure 33). Also in this feature, the correct accuracy of the classifier was computed with a another algorithm, which will be presented in the next paragraph.

45

Figure 33: Optimal window during the price response, 0.2:0.6 s

3.4 CLASSIFICATION

The goal of classification is to obtain an ordered set of related categories used to group data according to its similarities.

In the classification analysis is important to assess the reliability of the proposed predictor.

The method used in this work was the Two Fold Cross-Validation. It works using a defined training data, for which the class was known a priori, and a defined

validation data, for which the class was also known a priori and which is used to

determine the accuracy of the prediction.

The procedure was to apply the classification learned with the training data to the validation data and then to evaluate the classification accuracy rate using a supervised learning algorithms computed on training data.

46

The basic idea of Bootstrap is that inference about a population from sample data (sample → population) can be modeled byresamplingthe sample data and performing inference on (resample → sample). This method is recommended when the sample size is insufficient for straightforward statistical inference. A big advantage is the appropriate way to control and check the stability of the results. Having 30 shoes divided in 2 groups (bad and good), for the training data was chosen 20 shoes (10 randomly from the Good shoes group and 10 randomly from the Bad shoes group). The remaining shoes were assigned to the validation data. The possible combination of resample depends from the binomial coefficient (Equation 3.8):

𝑏𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝑐𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡 = (

) =

𝑘!(𝑛−𝑘)!

where 𝑛 is the number of the shoes in each group (𝑛=15) and 𝑘 is the number of shoes to select for the training set of each group (𝑘=10).

After the definition of the data, for each combination was applied the classification algorithm and all the obtained accuracies was averaged to better evaluate the real accuracy.

Different classifiers were implemented:

 Support Vector Machine

 Linear Discriminant Analysis

 Linear Classifier

 Nearest-Neighbor Classifier

The following paragraphs describe the classifiers adopted to better understand the theory behind and the operating principles of each classifier.

3.4.1 SUPPORT VECTOR MACHINE

The SVM classifier is capable of separating high-dimensional data. It separates the data from two classes by constructing a hyperplane between the training data points from both classes. It searches for the hyperplane that separates the two classes with

47

the largest distance to the nearest training data point of any class. The data points closest to the separating plane are called support vectors. This process is depicted in Figure 34.

Figure 34: Separation using Support Vector Machine

Having data points (𝑥1, 𝑐1), (𝑥2, 𝑐2), … (𝑥𝑛, 𝑐𝑛), where 𝑥𝑖 is the point to be classified

and 𝑐_𝑖 is the class to which the point belongs, -1 or 1. The SVM searches for a hyperplane to divide the two classes with the formula

𝑤 ∙ 𝑥 − 𝑏 = 0

where vector 𝑤 points perpendicular to the separating hyperplane and 𝑏 is the axis of ordinates intercept. If the two classes are linearly separable, the parameters 𝑤 and 𝑏 can be chosen such that

48

for all samples. This method is based on finding the hyperplane that separates the data points best, which is the one that keeps the largest margin between the data points. The closest data points to the margins are the support vectors, and two support planes are created through those data points:

𝑤 ∙ 𝑥

− 𝑏 = 1

𝑤 ∙ 𝑥

− 𝑏 = −1

The margin between the separating hyperplane and the two support planes is equal to 1/|𝑤|, so in order to maximize this distance we should minimize |𝑤|. When 𝑤 and 𝑏 are chosen also such that

𝑤 ∙ 𝑥

− 𝑏 > 1 𝑓𝑜𝑟 𝑐

= 1

𝑤 ∙ 𝑥

− 𝑏 < − 1 𝑓𝑜𝑟 𝑐

= −1

the problem can be rewritten to minimize |𝑤| subject to

𝑐

(𝑤 ∙ 𝑥

− 𝑏) ≥ 1

where 1 ≤ 𝑖 ≤ 𝑛, which can be solved using existing minimization algorithms. SVMs are very powerful classifiers. Even though it only searches for a separating hyperplane, it is capable of finding very complex divisions between classes. The power of this method lies in the fact that the data is transformed into a high-dimensional space, using a so-called kernel function. Using a proper transformation, it will be easier to separate the points in this higher dimensional space (Figure 35).

49

Figure 35: Support Vector Machine kernel function. (a) the 1-dimensional points cannot be linearly separated. By the transformation to the point (x, x2) a linear separation can be made (b)

However, the memory and computational demands increase drastically with a larger training set. This wasn’t a problem in this work, as the training set used was small.

3.4.2 LINEAR DISCRIMINANT ANALYSIS

LDA, also known as Fisher’s discriminant analysis, seeks to reduce dimensionality while preserving as much of the class discriminatory information as possible.

Having data points, 𝑥 ∈ 𝐷, which belong to two classes, 𝑤₁ and 𝑤₂, the method consists of finding a weights vector, 𝑤, such that

𝑤

_{𝑥 + 𝑤}

0

{> 0

_{< 0}

⇒ 𝑥 ∈ {

𝑤

𝑤

The Fisher’s criterion consists of searching for a linear combination of variables (predictors) that best separates the two classes (targets) maximizing the ratio of between-class variance to within-class variance (Equation 3.15) as illustrated by Figure 36.

50

𝐽

=

𝑤𝑇𝑆𝑤𝑤 Equation 3. 15

where 𝑚1 and 𝑚2 are the averages of the groups and 𝑆𝑤 is the within-class scatter

matrix, computed by the Equation 3.16

𝑆

= ∑

(∑

(𝑥 − 𝑚

)(𝑥 − 𝑚

)

𝑥∈𝐷

)

𝑐

𝑖 =1 Equation 3. 16

The optimal solution is obtained finding the maximum of 𝐽_𝐹, deriving and equaling to zero, as in Equation 3.17

𝑑

𝑑𝑤

[𝐽

] = 0

The optimal solutions are

𝑤 = 𝑆_𝑤−1_(𝑚 1− 𝑚2) with

𝑤

= −

(𝑚

+ 𝑚

)

_𝑆

(𝑚

− 𝑚

) − 𝑙𝑜𝑔 (

)

51

3.4.3 LINEAR CLASSIFIER

Figure 37: Example of Linear Classifier for two groups

The Figure 37 shows a Linear Classifier applied to data belonging to two different groups. The algorithm works by finding the average point (centroid) for each of the two labeled data sets, in this case the training data. It then draws a straight line between the centroids, the middle point of this first straight line is individuated and two segments, 𝑑₁ and 𝑑₂, are evaluated as half of the total distance, 𝑙, between the two average points, as in the Equation 3.19:

𝑑

=

2

=

√(𝑥₁−𝑥₂)2_{+(𝑦1−𝑦2)}2

2 Equation 3. 19

where (𝑥₁, 𝑦₁) and (𝑥₂, 𝑦₂) are the average points coordinates. The 𝑑₁ and 𝑑₂ segments are equal.

The separation line (black line in Figure 37) is made from points have the same distance from the centroids, where the condition 𝑑₁ = 𝑑₂ is still true. This line is the classifier and divides the two data sets.

d1