Hypothesis Testing & Power

In statistical inference, hypothesis testing is the formal process of using data to evaluate the validity of a claim about a population parameter. This lesson moves beyond introductory “plug-and-chug” methods to explore the mathematical foundations of decision theory, likelihood ratios, and the optimization of test power.

1. The Decision Framework: $H_0$ and $H_1$

We define two competing hypotheses:

• Null Hypothesis ($H_0$): The status quo or a specific “no effect” state. Mathematically, it typically specifies a subset of the parameter space ${\Theta }_0\subset \Theta$.
• Alternative Hypothesis ($H_1$): The statement we seek to find evidence for, ${\Theta }_1=\Theta \setminus {\Theta }_0$.

A test is a decision rule $\delta (X)$ that maps the sample space $\mathcal{X}$ to the set $\{H_0,H_1\}$. This is often defined via a rejection region $R\subset \mathcal{X}$:

2. Errors in Decision Making

Errors are unavoidable in frequentist inference. We quantify them as probabilities:

• Type I Error ($\alpha$): Rejecting $H_0$ when it is true.

  $\alpha =P(\text{Reject }{H}_0|H_0\text{ is true})=P(X\in R|\theta \in {\Theta }_0)$

• Type II Error ($\beta$): Failing to reject $H_0$ when it is false.

  $\beta =P(\text{Fail to reject }{H}_0|H_1\text{ is true})=P(X\notin R|\theta \in {\Theta }_1)$

• Power of the Test ($1-\beta$): The probability of correctly rejecting a false null hypothesis.

  $\text{Power}(\theta )=P(X\in R|\theta \in {\Theta }_1)$

Ideally, we minimize both $\alpha$ and $\beta$. However, for a fixed sample size $n$, there is an inverse relationship: decreasing the “size” ($\alpha$) of the test generally increases $\beta$.

3. Test Statistics and Rejection Regions

A test statistic $T(X)$ reduces the dimensionality of the data to a single value used for the decision. Common forms include:

The Z-Test (Known Variance)

Under $H_0:\mu ={\mu }_0$ with known variance ${\sigma }^2$: $Z=\frac{\bar{X}-{\mu }_0}{\sigma /\sqrt{n}}\sim \mathcal{N}(0,1)$

The T-Test (Unknown Variance)

If ${\sigma }^2$ is unknown and estimated by sample variance $s^2$: $t=\frac{\bar{X}-{\mu }_0}{s/\sqrt{n}}\sim t_{n-1}$

The Chi-Squared Test

For variance testing or goodness-of-fit: ${\chi }^2=\sum \frac{(O_i-E_i{)}^2}{E_i}\sim {\chi }_k^2$

4. The P-Value: A Measure of Evidence

The p-value is not the probability that $H_0$ is true. Rather, it is the probability of observing a test statistic at least as extreme as the one computed, assuming $H_0$ is true.

Formally, for a test statistic $T(X)$ where large values provide evidence against $H_0$: $p=P(T\ge t_{obs}|H_0)$

A p-value is a random variable itself. If $H_0$ is true, the p-value is uniformly distributed on $[0,1]$ for continuous test statistics: $p\sim U(0,1)$.

5. The Neyman-Pearson Lemma

How do we choose the best rejection region $R$? For a simple null $H_0:\theta ={\theta }_0$ versus a simple alternative $H_1:\theta ={\theta }_1$, the Neyman-Pearson Lemma provides the Most Powerful (MP) test.

The lemma states that the region $R$ that maximizes power for a fixed $\alpha$ is defined by the Likelihood Ratio: $\Lambda (x)=\frac{L({\theta }_0|x)}{L({\theta }_1|x)}\le k$ where $k$ is chosen such that $P(\Lambda (X)\le k|{\theta }_0)=\alpha$. This ratio ensures that we reject $H_0$ when the data is significantly “more likely” under $H_1$ than under $H_0$.

6. Uniformly Most Powerful (UMP) Tests

When $H_1$ is composite (e.g., $\theta >{\theta }_0$), we seek a test that is the most powerful for all $\theta \in {\Theta }_1$. Such a test is called Uniformly Most Powerful (UMP).

The existence of a UMP test is guaranteed if the family of distributions possesses the Monotone Likelihood Ratio (MLR) property. A family $f(x|\theta )$ has MLR in $T(x)$ if for any ${\theta }_1<{\theta }_2$, the ratio $f(x|{\theta }_2)/f(x|{\theta }_1)$ is a non-decreasing function of $T(x)$.

7. Likelihood Ratio Tests (LRT) and Wilks’ Theorem

For complex, multi-parameter composite hypotheses, we use the generalized Likelihood Ratio Test: $\lambda (x)=\frac{{\sup }_{\theta \in {\Theta }_0}L(\theta |x)}{{\sup }_{\theta \in \Theta }L(\theta |x)}$ where $0\le \lambda \le 1$. Small values of $\lambda$ lead to rejection.

Wilks’ Theorem: Under certain regularity conditions, as $n\to \infty$, the distribution of $-2\ln \lambda (x)$ converges in distribution to a ${\chi }^2$ distribution with degrees of freedom equal to the difference in dimensionality between $\Theta$ and ${\Theta }_0$: $-2\ln \lambda (x)\stackrel{d}{\to }{\chi }_r^2$

8. Multiple Testing and Bonferroni Correction

When conducting $m$ independent tests at a significance level $\alpha$, the probability of committing at least one Type I error (Family-Wide Error Rate, FWER) is $1-(1-\alpha {)}^m$. As $m$ grows, this approaches 1.

The Bonferroni correction guards against this by using a stricter threshold for each individual test: ${\alpha }^{\prime }=\frac{\alpha }{m}$

Python Implementation: T-Test and Visualization

The following code calculates a one-sample t-test and visualizes the rejection region vs. the p-value.

import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# Parameters
mu_null = 50
sample_size = 30
data = np.random.normal(52, 10, sample_size) # Mean 52, StdDev 10

# Perform t-test
t_stat, p_val = stats.ttest_1samp(data, mu_null)
df = sample_size - 1

# Plotting
x = np.linspace(-4, 4, 1000)
y = stats.t.pdf(x, df)
critical_value = stats.t.ppf(0.95, df)

plt.figure(figsize=(10, 6))
plt.plot(x, y, label=f't-distribution (df={df})')

# Rejection Region (Alpha = 0.05)
plt.fill_between(x, 0, y, where=(x > critical_value), color='red', alpha=0.3, label='Rejection Region')

# P-value area
plt.fill_between(x, 0, y, where=(x > t_stat), color='blue', alpha=0.5, label=f'p-value area (p={p_val:.4f})')

plt.axvline(t_stat, color='black', linestyle='--', label=f'Observed t={t_stat:.2f}')
plt.title('One-Sample T-Test: Rejection Region vs P-value')
plt.legend()
plt.show()