Inferential Statistics Multiple tests and other useful amenities

(1)

Inferential Statistics

Multiple tests and other useful amenities

Eva Riccomagno, Maria Piera Rogantin

DIMA – Universit`a di Genova

riccomagno@dima.unige.it rogantin@dima.unige.it

(2)

Review

Part D Test of the equality of two means (two-samples and paired samples)

Part E Multiple tests

Part F Confidence intervals

(3)

Review

Exercise. Chicago Tribune. Chicago land’s technology professionals get local technology news from various newspapers and magazines. A marketing company claims that 25% of the IT professionals choose the Chicago Tribune as their primary source for local IT news. A survey was conducted last month to check this claim. Among a sample of 750 IT professionals in the Chicago land area, 23.43% of them prefer the Chicago Tribune. Can we conclude that the claim of the marketing company is true?

The random variable modeling the preference of the Chicago Tribune is X ∼ B(1, p).

Test statistic: P = X; sample value: ˆ^b p = 0.2343.

Large sample size (n = 750). Using CLT P ∼ N^b p, ^p(1−p)_n . H₀ : p = 0.25 which H₁? : p 6= 0.25 or p < 0.25??

(4)

p-value computation in R using t.test.

> prop.test(np,750,0.25)

1-sample proportions test with continuity correction data: np out of 750, null probability 0.25

X-squared = 0.904, df = 1, p-value = 0.3417

alternative hypothesis: true p is not equal to 0.25 95 percent confidence interval:

0.2047542 0.2666131 sample estimates:

p 0.2343

> prop.test(np,750,0.25,"less")

alternative hypothesis: true p is less than 0.25 95 percent confidence interval:

p 0.2343

In both cases there is not evidence to reject H₀ (p = 0.25)

(5)

Part D. Test for the equality of two means

A common application is to test if a new process or treatment is superior to a current process or treatment.

The data may either be paired or unpaired.

a) Paired samples When there is a one-to-one correspondence between the values in the two samples. That is, if X₁, X₂, . . . , X_n and Y₁, Y₂, . . . , Y_n are the two sample variables, then X_i cor- responds to Y_i.

b) Unpaired samples The sample sizes for the two samples may or may not be equal.

(6)

a) Paired samples

Let X and Y be two random variables modeling some character- istic of the same population.

Example. Drinking Water

(from https://onlinecourses.science.psu.edu Penny State University)

Trace metals in drinking water affect the flavor and an unusually high concentration can pose a health hazard.

Ten pairs of data were taken measuring zinc concentration in bottom water and surface water.

> water

bottom surface [1,] 0.430 0.415 [2,] 0.266 0.238 [3,] 0.567 0.390 [4,] 0.531 0.410 [5,] 0.707 0.605 [6,] 0.716 0.609 [7,] 0.651 0.632 [8,] 0.589 0.523 [9,] 0.469 0.411 [10,] 0.723 0.612

(7)

Assume X ∼ N (µ_X, σ_X² ) and Y ∼ N (µ_Y , σ_Y²).

Th test hypotheses are:

H₀ : µ_X = µ_Y and H₁ : µ_X 6= µ_Y

or equivalently: H₀ : µ_X − µ_Y = 0 and H₁ : µ_X − µ_Y 6= 0 H₁ could be µ_X < µ_Y or µ_X > µ_Y and possibly H₀ composite.

Let (X₁, Y₁), . . . , (X_n, Y_n) be the n paired sample variables.

Consider the sample random variables D₁, . . . , D_n with D_i = X_i − Y_i.

Consider the sample mean of D:

D ∼ N (µ_D, σ_D² /n)

with µ_D = µ_X − µ_Y and σ_D² = σ_X² + σ_Y² − 2Cov(X, Y ), usually unknown and estimated by the unbiased estimator S_D² .

The test of the equality of the two means becomes a Student’s t test on µ_D, with H₀ : µ_D = 0.

(8)

Example. Drinking Water (continue)

> D=water[,1]-water[,2];D

[1] 0.015 0.028 0.177 0.121 0.102 0.107 0.019 0.066 0.058 0.111

• Hypotheses: H₀ : µ_D = 0 and H₁ : µ_D 6= 0

• Two-sided: R₀ = (−∞, c₁) ∪ (c₂, +∞)

• Sample size: n = 10

• Sample variables: D₁, . . . , D₂₅ i.i.d. D_i ∼ N (0, σ_D² ) with σ_D² estimated by S_D²

• Test statistic under H₀: T = ^D

S_D/√

n ∼ t₉

• α = 0.05

The thresholds of the rejection region c₁ and c₂ are such that 0.025 = P(T < c1 | µ_D = 0) 0.025 = P(T > c2 | µ_D = 0) Observe that, because of the symmetry w.r.t. 0 of the Student’s t density

c₁ = −c₂

(9)

In the sample: d = 0.0804 s = 0.052

The sample value of the test statistic, under H0, is 4.86

> d_m=mean(D);d_m; s=sd(D);s [1] 0.0804

[1] 0.05227321

> t=d_m/(s/sqrt(10));t [1] 4.863813

The rejection region is R0 = (−∞, −2.262) ∪ (2.262, ∞). The p-value is 0.0009

> c1=qt(0.025,9) [[1] -2.262157

> 2*(1-pt(t,9)) [1] 0.0008911155

The direct computation in R produces:

> t.test(water[,1],water[,2],paired=TRUE) Paired t-test

data: water[, 1] and water[, 2]

t = 4.8638, df = 9, p-value = 0.0008911

alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval:

mean of the differences 0.0804

There is experimental evidence to reject H0

(10)

b) Unpaired samples Example. Prey of two species of spiders

(from https://onlinecourses.science.psu.edu Penny State University)

The feeding habits of two species of net-casting spiders are stud- ied. The species, the deinopis and menneus, coexist in eastern Australia. The following data were obtained on the size, in mil- limeters, of the prey of random samples of the two species.

The spiders were selected randomly. Then assume the measure- ments are independent.

> d=c(12.9,10.2,7.4,7.0,10.5,11.9,7.1,9.9,4.4,11.3)

> m=c(10.2,6.9,10.9,11.0,10.1,5.3,7.5,10.3,9.2,8.8)

> mean(d);mean(m) [1] 10.26

[1] 9.02

d m

68101214

(11)

Normal distribution

Assume the size of the two population (denoted by A and B) have normal distribution

X_A ∼ N (µ_A, σ_A²) X_B ∼ N (µ_B, σ_B² )

We want to test H₀ : µ_A = µ_B and H₁ : µ_A 6= µ_B

or equivalently: H₀ : µ_A − µ_B = 0 and H₁ : µ_A − µ_B 6= 0

Let n_A and n_B be the size of the two independent sample of X_A and X_B. In the example n_A = n_B = 10.

The two sample mean random variable are:

X_A ∼ N µ_A, σ_A² n_A

!

X_B ∼ N µ_B, σ_B² n_B

!

Consider the random variable difference of the two sample mean random variables. It has normal distribution:

X_A − X_B ∼ N µ_A − µ_B, σ_A²

n_A + σ_B² n_B

!

The original test become a test on the mean of one normal random variable

(12)

1. The variances σ_A² and σ_B² are known

Fixed α, a usual normal test is carried out.

2. The variances σ_A² and σ_B² are unknown, and assumed equal and estimated by the unbiased estimators S_A² e S_B².

A unbiased estimator of the variance of the random variable X_A − X_B is:

S² = (n_A − 1)S_A² + (n_B − 1)S_B²

(n_A + n_B − 2) · n_A + n_B

n_A n_B (Pooled variance) In particular, if n_A = n_B = n, then S² = S_A² + S_B²

/n

The test statistic is:

T =

X_A − X_B − (µ_A − µ_B)

S with T ∼ t_d d = n_A + n_B − 2 Fixed α, a usual Student’s t test is carried out.

3. The unknown variances σ_A² and σ_A² are not equal

A hypothesis test based on the t distribution, known as Welch’s t-test, can be used.

(13)

Example. Prey of two species of spiders (continue)

• Hypotheses: H₀ : µ_D = µ_M and H₁ : µ_D 6= µ_M

• Two-sided: R₀ = (−∞, c₁) ∪ (c₂, +∞)

• Sample size: n_D = n_M = 10

• First, assume σ_M² = σ_D² . Pooled variance estimator:

S² = S_D² + S_M² /n and d = 2n − 2

• Test statistic under H₀: T =

X_D − X_M

S ∼ t₁₈

• α = 0.05

The thresholds of the rejection region c₁ and c₂ are such that 0.025 = P(T < c1 | µ_D = µ_M) 0.025 = P(T > c2 | µ_D = µ_M)

(14)

The sample means of the two groups are:

x_D = 10.26 x_M = 9.0.2

The sample difference of means is: x_D − x_M = 1.24 The sample pooled variance is: s² = 1.01

The sample value of the test statistic, under H₀, is 1.18

> diff_m=mean(d)-mean(m);diff_m [1] 1.24

> s2=(sd(d)^2+sd(m)^2)/10;s2 [1] 0.9915556

> t=diff_m/sqrt(s2);t [1] 1.245269

The rejection region is R₀ = (−∞, −2.1) ∪ (2.1, ∞).

The p-value is 0.25

> c1=qt(0.025,18);c1 [1] -2.100922

> 2*(1-pt(t,18)) ## note 2*( ) -- two sided test [1] 0.2290008

There is no experimental evidence to reject H₀

(15)

Can we assume the two variances equal?

A specific test can be performed (based on the Fisher distribution). Here we do not give the details. Compute in R

> var.test(m, d, ratio = 1)

F test to compare two variances data: m and d

F = 0.56936, num df = 9, denom df = 9, p-value = 0.4142

alternative hypothesis: true ratio of variances is not equal to 1 95 percent confidence interval:

ratio of variances 0.5693585

We can assume the σ_D² = σ_M² , although the ratio of variances is 0.57. This apparent inconsistency is due to the small sample sizes.

Direct computation in R of the test

H₀ : µ_D = µ_M and H₁ : µ_D 6= µ_M, assuming σ_D² = σ_M²

> t.test(d,m,var.equal=T) Two Sample t-test data: d and m

t = 1.2453, df = 18, p-value = 0.229

-0.8520327 3.3320327 sample estimates:

mean of x mean of y 10.26 9.02

(16)

If the equality of the variances is rejected, we use the Welch Two Sample t-test.

In such a case the polled variance s² and the degrees of freedom are compute in an another manner.

Compute in R

t.test(d,m)

Welch Two Sample t-test data: d and m

t = 1.2453, df = 16.74, p-value = 0.2302

-0.8633815 3.3433815 sample estimates:

mean of x mean of y 10.26 9.02

The problem of making inference on means when variances are unequal, is, in general, quite a difficult one. It is known as the Behrens-Fisher Problem.

(G. Casella, R.J. Berger, Statistical Inference, 2nd ed., Duxbury, Ex.

8.42)

(17)

Notes and generalisations

• The Wald test. If the two random variables have not normal distribution and the sample size is “large” a Wald test can be performed.

• Threshold different from zero. In some applications, you may want to adopt a new process or treatment only if it exceeds the current treatment by some threshold. In this case, the difference between the two mean is not compared with 0 but with the chosen threshold.

(18)

Part E. Multiple tests

We may need to conduct many hypothesis tests with the same data.

Suppose each test is conducted at level α.

For any one test, the chance of a false rejection of the null is α.

But the chance of at least one false rejection is much higher.

Examples:

• Measuring the state of anxiety by questionnaire in two groups of subjects. Various questions help define the level of anxiety.

As more topics are compared, it becomes more likely that the two groups will appear to differ on at least one topic by random chance alone.

• Efficacy of a drug in terms of the reduction of any one of a number of disease symptoms. It becomes more likely that the drug will appear to be an improvement over existing drugs in terms of at least one symptom.

(19)

Consider m independent hypothesis tests:

H₀ⁱ and H₁ⁱ for i = 1, . . . , m Example.

For α = 0.05 and m = 2

Probability to retain both H₀¹ and H₀² when true: (1 − α)² = 0.95² = 0.90 Probability to reject at least one true hypothesis: 1−(1−α)² = 1−0.95² = 0.10

For α = 0.05 and m = 20

Probability to reject at least one true hypothesis:

1 − (1 − α)²⁰ = 1 − 0.95²⁰ = 0.64 α

There are many ways to deal with this problem. Here we discuss two methods.

(20)

Bonferroni (B) Method

Given p-values p₁, . . . , p_m, reject null hypothesis H₀ⁱ if p_i ≤ α

m

The probability of falsely rejecting any null hypotheses is less than or equal to α.

Example (continue)

m = 2: 1 − (1 − (0.05/2))² = 0.0493 m = 20: 1 − (1 − (0.05/20))²⁰ = 0.0488

Benjamini-Hochberg (BH) Method

1. Let p₍₁₎ < · · · < p_(m) denote the ordered p-values.

2. Reject all null hypotheses H₀⁽ⁱ⁾ for which p_(i) < _αⁱ m

If the tests are not independent the value to compare p_(i) is appropriately adjusted.

(21)

Example

Consider the following 10 (sorted) p-values. Fix α = 0.05.

p=c(0.00017,0.00448,0.00671,0.00907,0.01220,0.33626,0.39341, 0.53882,0.58125,0.98617)

alpha=0.05; m=length(p); i=seq(1,m);

b=i/m*alpha; BH=(p<b);

B=(p<alpha/m); cbind(p,b,BH,B) p b BH B

[1,] 0.00017 0.005 1 1 [2,] 0.00448 0.010 1 1 [3,] 0.00671 0.015 1 0 [4,] 0.00907 0.020 1 0 [5,] 0.01220 0.025 1 0 [6,] 0.33626 0.030 0 0 [7,] 0.39341 0.035 0 0 [8,] 0.53882 0.040 0 0 [9,] 0.58125 0.045 0 0 [10,] 0.98617 0.050 0 0

Reject H₀ⁱ for

- i = 1, 2 with Bonferroni method

- i = 1, 2, 3, 4, 5 with Benjamini-Hochberg method.

(22)

Abuse of test

Warning! There is a tendency to use hypothesis testing methods even when they are not appropriate. Often, estimation and confidence intervals are better tools. Use hypothesis testing only when you want to test a well-defined hypothesis.

(from Wassermann)

(23)

A summary of the paper by Regina Nuzzo. (2014) Statistical Errors – P values, the “gold standard” of statistical validity, are not as reliable as many scientists assume. Nature, vol. 506, p. 150-2.

• Ronald Fisher 1920s

– intended as an informal way to judge whether evidence was significant for a second look

– one part of a fluid, non-numerical process that blended data and background knowledge to lead to scientific con- clusions

• Interpretation

– the P value summarizes the data assuming a specific null hypothesis

(24)

• Caveats

– tendency to deflect attention from the actual size of an effect

– P-hacking or significance-chasing, including making as- sumptions, monitor data while it is being collected, ex- cluding data points, ...

• Measures that can help – look for replicability

– do not ignore you exploratory studies nor prior knowledge – report effect sizes and confidence intervals

– take advantage of Bayes’ rule (not part of this course unfortunately)

– try multiple methods on the same data set

– adopt a two-stage analysis, or “preregistered replication”

(25)

Part F

Confidence intervals 1. Introduction

2. Confidence interval for the mean

(a) of a Normal variable – known variance (b) of a Normal variable – unknown variance

(c) of a variable with unknown distribution (approximate) 3. Different levels 1 − α

4. Asymmetric and one-sided intervals 5. Confidence intervals and tests

(26)

1. Introduction

A 1− α confidence interval for a parameter θ is an interval (L, U ) where L and U are functions of the random sample X₁, . . . , X_n (i.e. are random variables) such that

P (θ ∈ (L, U )) ≥ 1 − α for all θ ∈ Θ The interval (L, U ) is also called interval estimator.

1 − α is called the coverage of the confidence interval. Usually 1 − α = 0.95

We have over a 1− α chance of covering the unknown parameter with the estimator interval (from Casella Berger).

(27)

From Wassermann.

Warning! (L, U ) is random and θ is fixed.

Warning! There is much confusion about how to interpret a confidence interval. A confidence interval is not a probability statement about θ since θ is a fixed quantity [. . . ].

Warning! Some texts interpret confidence intervals as follows: if I repeat the experiment over and over, the interval will contain the parameter 95 percent of the time. This is correct but useless since we rarely repeat the same experiment over and over. [. . . ] Rather

- day 1: θ₁ ⇒ collect data ⇒ construct a 95% IC for θ₁ - day 2: θ₂ ⇒ collect data ⇒ construct a 95% IC for θ₂ - day 3: θ₃ ⇒ collect data ⇒ construct a 95% IC for θ₃ - . . .

Then 95 percent of your intervals will trap the true parameter value. There is no need to introduce the idea of repeating the same experiment over and over.

(28)

2. Confidence interval for the mean of a random variable Let X₁, . . . , X_n be an i.i.d. random sample. Parameter of interest µ.

• Point estimator: X;

point estimate x (sample value of X at the observed data points)

• Confidence interval or Interval estimator with coverage 1− α:

X − δ , X + δ with δ such that P

X − δ < µ < X + δ = 1 − α

The limit of the interval X − δ and X + δ are random variables The sample confidence interval is:

(x − δ, x + δ)

How to compute δ? Using the (exact or approximate) distribution of the point estimator X.

(29)

2. (a) Confidence interval for the mean of a Normal variable – known variance

For X₁, . . . , X_n i.i.d. sample random variables with X₁ ∼ N (µ, σ²) X ∼ N µ, σ²

n

!

or Z = X − µ σ/√

n ∼ N (0, 1)) 1 − α = PX − δ < µ < X + δ = P

µ − δ < X < µ + δ

Example.

X₁ ∼ N (µ, 4) n = 9 1−α = 0.95

X ∼ N

µ, 4 9

0.00.10.20.30.40.50.60.00.10.20.30.40.50.6

µ − δ µ µ + δ

(30)

Computation of δ

1 − α = P µ − δ < X < µ + δ

= P µ − δ − µ

√σ n

< X − µ

√σ n

< µ + δ − µ

√σ n

!

= P − δ

√σ n

< Z < δ

√σ n

!

⇒ δ

√σ n

= z_1−α/2 ⇒ δ = z_1−α/2 σ

√n

Density functions of - Z ∼ N (0, 1)

- X ∼ N (µ, 4/9) z_1−0.05/2 = 1.96 δ = 1.31

0.00.10.20.30.40.50.60.00.10.20.30.40.50.6

µ − δ µ µ + δ

0.00.10.20.30.40.50.60.00.10.20.30.40.50.6

−1.96 0 1.96

Confidence interval for µ^: X − z_1−α/2 σ

√n, X + z_1−α/2 σ

√n

!

(31)

Sample confidence interval for µ:

x − z_1−α/2 σ

√n, x + z_1−α/2 σ

√n

!

- we do not know if µ belongs or not to this sample interval whose limits are computed using the sample value x

- whit another x the interval would be different

Among all possible confidence intervals constructed as before, 95% contains µ and 5% does not.

Simulation for 100 samples: n = 80 σ² = 4 1 − α = 95%

x − 1.96 2/√

80, x + 1.96 2/√

80

6 intervals do not contain µ.

(32)

2. (b) Confidence interval for the mean of a Normal variable – unknown variance

For X₁, . . . , X_n random sample and X₁ ∼ N (µ, σ²) as point estimator of µ and σ² take X and S² respectively.

Consider the random variable

T = X − µ S/√

n ∼ t_[n−1]

The computation of the confidence interval for µ is similar to the normal case

X − t_1−α/2 S

√n, X + t_1−α/2 S

√n

!

2. (c) Confidence interval for the mean of a random variable with unknown distribution

If the sample size is “large” we can use the approximate distribution of X via CLT:

X − z_1−α/2 S

√n, X + z_1−α/2 S

√n

!

(33)

Examples

Consider an i.i.d random sample and fix α = 0.05

• n = 36

– X₁ ∼ N (µ, 4); confidence interval: (¯x − 1.96 2/6, ....)

– X₁ ∼ N (µ, σ²), s = 2; confidence interval: (¯x−2.03 2/6, ....)

• n = 100

– X₁ ∼ N (µ, 4); confidence interval: (¯x − 1.96 2/10, ....)

– X₁ ∼ N (µ, σ²), s = 2; confidence interval: (¯x−1.98 2/10, ....) where ¯x and s obtained from the same data.

(34)

3. Different coverage coefficients 1 − α A 95%-confidence interval or a 99%-confidence interval?

Values of the 1 − α/2 quantile of a standard normal random variable N (0, 1): z_0.950 = 1.64 z_0.975 = 1.96 z_0.995 = 2.58

0.95 0.99 0.90

What is gained in precision is lost in range

Example X ∼ N (µ, 4/80) and assume x = 2.5:

- at 90%: δ = 0.37 sample confidence interval (1.92, 3.08) - at 95%: δ = 0.44 sample confidence interval (2.06, 2.94) - at 99%: δ = 0.58 sample confidence interval (2.13, 2.87)

(35)

4. Asymmetric and one-sided intervals

When the distribution of the point estimator used to compute the limits of the confidence interval is symmetric w.r.t. the parameter of interest, it is natural to consider “symmetric” intervals, e.g.

(X − δ, X + δ).

This is not the case, for instance, when parameter of interest is σ². The confidence interval is (we do not give the details):

S² n − 1

q_r , S² n − 1 q_l

!

where q_l is the quantile α_l and q_r is the quantile 1 − α_r of a Chi square random variable with n − 1 degrees of freedom, with

α_l + α_r = α non necessarily α_l = α_r = α/2

(36)

Examples n = 5 S^{2 n−1}_q

r , S^{2 n−1}_q

l

α_l = 0.03, α_r = 0.02

n−1

q_r = 0.3 ⁿ⁻¹_q

l = 4.4

Asymmetric confidence interval for σ²

0.3 S², 4.4 S²

α_l = 0.05 ⁿ⁻¹_q

l = 3.5

One-sided confidence interval for σ²

0, 3.5 S²

0 5 10 15

0.000.050.100.15

0 5 10 15

0.000.050.100.15

0 5 10 15

0.000.050.100.15

q_l q_r

0 5 10 15

0.000.050.100.15

0 5 10 15

0.000.050.100.15

q_l

(37)

5. Confidence intervals and tests Parameter of interest µ of a N (µ, σ) with σ known

Two-sided 1 − α confidence interval for µ ^and two-sided test at level α (H₀ : µ = µ₀ and H₁ : µ 6= µ₀)

H₀ is retained for x ∈ (µ₀ − δ, µ₀ + δ) The sample confidence interval is (x − δ, x + δ)

( µ )

( x

^A

) ( x

^B

)

0

where δ = z_1−α/2 ^√^σ

n for both confidence interval and test.

The interval where H₀ is retained is centered in µ₀ while the confidence interval is centered in x

If the sample confidence interval contains µ₀ then H₀ is retained, and viceversa.

(38)

One-side right confidence interval and one-sided left test

H₀ is retained for x ∈ (µ₀ − δ₁, +∞) The sample left confidence interval is (∞, x + δ₁)

x

^A

( µ

)

0

( )

where δ₁ = z_1−α ^√^σ

n for both confidence interval and test.

Remark.

If the parameter of interest is a proportion p then δ is different for confidence intervals and tests, because it depends on the standard deviation. In the first case it is calculated using the sample value ˆp, in the second one using p₀.

(39)

Compare tests and confidence intervals in output R

Example. Chicago Tribune (continue)

> prop.test(np,750,0.25)

alternative hypothesis: true p is not equal to 0.25 95 percent confidence interval:

p 0.2343

> prop.test(np,750,0.25,"less")

alternative hypothesis: true p is less than 0.25 95 percent confidence interval:

p 0.2343