Exploratory Data Analysis part 3

Exploratory Data Analysis part 3

Eva Riccomagno, Maria Piera Rogantin

DIMA – Universit`a di Genova

http://www.dima.unige.it/~rogantin/UnigeStat/

Indices for quantitative variables

Position indices – central tendencies

- median: Q2 = min{x | F (x) ≥ 0.50}

- mean: x = ¹_n ^Pn_i=1 x_i

- trimmed mean: mean of the 90% of the “central” data - mode: value with maximal frequency

1. ^Pn_i=1 (x_i − x) = 0

2. ^Pn_i=1 (x_i − x)² ≤ ^Pn_i=1 (x_i − a)² per ogni a ∈ R - the mean is centroid of

the distribution (equilibrium point)

- the mean is affected by out- liers, the median is not

60 80 100 120 140

● ● ●●

●

●

● ●●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

● ● ●●

● ● ● ● ● ●● ●●

● ●

3. ^Pn_i=1 |x_i − Q2| ≤ ^Pn_i=1|x_i − a| for all a ∈ R

Variability indices

• ranges

- range (R): max - min - interquartile range (IQR): Q3−Q1

• indices based on the deviations from a central value - variance and standard deviation

(V(X) or σ_X² or σ² – std(X) or σ_X or σ)

V(X) = 1 n

n X

i=1

(x_i − x)² std(X) =

v u u t

1 n

n X

i=1

(x_i − x)²

(Variance and standard deviation can be defined with n − 1)

- mean absolute deviations (from mean and median) 1

n

n X

i=1

|x_i − x| 1 n

n X

i=1

|x_i − Q2|

• variability w.r.t. central value coefficient of variation CV(X) = std(X)

x (if x 6= 0).

Properties: σ ≤ ^R₂ and |x − Q2| < σ

Mean and variance in a population and its sub-groups

Two subgroups: A and B

n_A and n_B units, f_A and f_B frequencies

x_A and x_B means and σ_A² and σ_B² variances

Frequency

10 20 30 40 50

010203040

Frequency

10 20 30 40 50

010203040

n_A = 100, x_A = 9.9, σ_A² = 6.5; n_B = 300, x_B = 30.0, σ_B² = 4.4 Mean

x_tot = f_A x_A + f_B x_B (weighted mean) x_tot = 100

9.9 + 300

30.0 = 24.98

Variance

Frequency

10 20 30 40 50

010203040

Frequency

10 20 30 40 50

010203040

x_B = 30.0

Frequency

10 20 30 40 50

010203040

Frequency

10 20 30 40 50

010203040

x_B = 40.0

σ_tot² = ^f_A σ_A² + f_B σ_B^{2 } + ^f_A (x_A − x_tot)² + f_B (x_B − x_tot)^2 weighted variance plus weighted “variance of the means”

In the example:

above: σ_tot² = 80.68 below: σ_tot² = 179.53

In general

x_tot =

K X

k=1

f_k x_k

σ_tot² =

K X

k=1

f_k σ_k² +

K X

k=1

f_k (x_k − x_tot)²

total variance =

within classes variance + between classes variance

R code for white, red and blue histograms

a=rnorm(100,10,2.4);mean(a);var(a) b=rnorm(300,30,2);mean(b);var(b) c=b+10

br=seq(1,50,.5)

hist(a, breaks=br,main=" ", xlab=" ",xlim=c(5,50),ylim=c(0,40)) par(new=T)

hist(b, breaks=br,main=" ", xlab=" ",xlim=c(5,50),ylim=c(0,40),col="red") par(new=F)

hist(a, breaks=br,main=" ", xlab=" ",xlim=c(5,50),ylim=c(0,40)) par(new=T)

hist(c, breaks=br,main=" ", xlab=" ",xlim=c(5,50),ylim=c(0,40),col="blue")