Statistical Inference & Estimation

Statistical inference is the process of using data analysis to deduce properties of an underlying distribution of probability. We assume that the observed data $x=(x_1,x_2,\dots ,x_n)$ are realizations of random variables $X_1,X_2,\dots ,X_n$ distributed according to some member of a parametric family $\mathcal{P}=\{f(x;\theta ):\theta \in \Theta \}$.

1. Point Estimation

A point estimator $\hat{\theta }$ is a statistic (a function of the data) used to approximate the unknown parameter $\theta$.

Bias and Mean Squared Error (MSE)

The Bias of an estimator $\hat{\theta }$ is defined as: $B(\hat{\theta })=E_{\theta }[\hat{\theta }]-\theta$ An estimator is unbiased if $B(\hat{\theta })=0$.

The Mean Squared Error (MSE) measures the average squared difference between the estimator and the parameter: $MSE(\hat{\theta })=E_{\theta }[(\hat{\theta }-\theta {)}^2]$ A fundamental decomposition of MSE is: $MSE(\hat{\theta })=Va{r}_{\theta }(\hat{\theta })+[B(\hat{\theta }){]}^2$ This highlights the bias-variance tradeoff: as we reduce bias, variance often increases, and vice-versa.

Consistency

An estimator ${\hat{\theta }}_n$ is consistent if it converges in probability to the true parameter: $\forall ϵ>0:{\lim }_{n\to \infty }{P}_{\theta }(|{\hat{\theta }}_n-\theta |>ϵ)=0$ This is often denoted as ${\hat{\theta }}_n\stackrel{P}{\to }\theta$. By the Law of Large Numbers, the sample mean $\bar{X}$ is a consistent estimator of the population mean $\mu$.

2. Maximum Likelihood Estimation (MLE)

MLE is the most widely used method for point estimation. Given i.i.d. observations, the Likelihood Function $L(\theta )$ is: $L(\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$ We seek ${\hat{\theta }}_{MLE}$ that maximizes $L(\theta )$. In practice, it is easier to maximize the log-likelihood: $\ell (\theta )=\ln L(\theta )={\sum }_{i=1}^n\ln f(x_i;\theta )$

The Score Function

The Score Function $U(\theta )$ is the gradient of the log-likelihood: $U(\theta )={\nabla }_{\theta }\ell (\theta )$ The MLE is found by solving the likelihood equation $U(\hat{\theta })=0$.

Asymptotic Properties of MLE

Under “regularity conditions,” MLEs possess desirable large-sample properties:

1. Consistency: ${\hat{\theta }}_{MLE}\stackrel{P}{\to }{\theta }_0$.
2. Asymptotic Normality: $\sqrt{n}({\hat{\theta }}_{MLE}-{\theta }_0)\stackrel{d}{\to }N(0,I(\theta {)}^{-1})$, where $I(\theta )$ is the Fisher Information.
3. Efficiency: For large $n$, MLE achieves the Cramér-Rao Lower Bound.

3. Method of Moments (MoM)

MoM estimates parameters by equating population moments to sample moments. If ${\mu }_k(\theta )=E[X^k]$, we solve the system: ${\mu }_k(\theta )=\frac{1}{n}{\sum }_{i=1}^n{x}_i^k,k=1,\dots ,p$ MoM is often easier to compute than MLE but is usually less efficient (higher variance).

4. Sufficient Statistics

A statistic $T(X)$ is sufficient for $\theta$ if the conditional distribution of $X$ given $T(X)$ does not depend on $\theta$. This means $T(X)$ captures all the information in the sample about $\theta$.

Factorization Theorem (Fisher-Neyman)

$T(X)$ is sufficient for $\theta$ if and only if the joint density can be factored as: $f(x;\theta )=h(x)g(T(x);\theta )$ where $h(x)$ does not depend on $\theta$ and $g$ depends on $x$ only through $T(x)$.

5. Information & Efficiency

Fisher Information

The Fisher Information $I(\theta )$ represents the amount of information that an observable random variable $X$ carries about an unknown parameter $\theta$: $I(\theta )=E_{\theta }\left[{\left(\frac{\partial }{\partial \theta }\ln f(X;\theta )\right)}^2\right]=-E_{\theta }\left[\frac{{\partial }^2}{\partial {\theta }^2}\ln f(X;\theta )\right]$

Cramér-Rao Lower Bound (CRLB)

For any unbiased estimator $\hat{\theta }$, its variance is bounded from below: $Var(\hat{\theta })\ge \frac{1}{nI(\theta )}$ An unbiased estimator that achieves this bound is called UMVUE (Uniformly Minimum Variance Unbiased Estimator) if it is efficient.

Rao-Blackwell Theorem

If $\hat{\theta }$ is an unbiased estimator and $T$ is a sufficient statistic, then the conditional expectation ${\hat{\theta }}^{\ast }=E[\hat{\theta }|T]$ is also unbiased and $Var({\hat{\theta }}^{\ast })\le Var(\hat{\theta })$. This implies that we only ever need to search for optimal estimators among functions of sufficient statistics.

6. Interval Estimation

Instead of a single point, we construct a Confidence Interval (CI) $[L,U]$ such that: $P_{\theta }(L(X)\le \theta \le U(X))=1-\alpha$ This is often done using a Pivotal Quantity $Q(X,\theta )$, a function of the data and the parameter whose distribution does not depend on $\theta$. Example: For $X\sim N(\mu ,{\sigma }^2)$ with known $\sigma$, $Z=\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}$ is a pivot since $Z\sim N(0,1)$.

Python Implementation: MLE for Poisson Distribution

The following code calculates the MLE for the $\lambda$ parameter of a Poisson distribution and visualizes the log-likelihood surface.

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize_scalar
from scipy.stats import poisson

# Generate synthetic Poisson data with true lambda = 4.5
np.random.seed(42)
true_lambda = 4.5
data = np.random.poisson(true_lambda, size=100)

def log_likelihood(lam, data):
    if lam <= 0: return -np.inf
    # Poisson PMF: (lam^k * e^-lam) / k!
    return np.sum(poisson.logpmf(data, lam))

# We want to maximize log-likelihood, which is minimizing the negative log-likelihood
def neg_log_likelihood(lam, data):
    return -log_likelihood(lam, data)

# Find MLE using scipy
res = minimize_scalar(neg_log_likelihood, args=(data,), bounds=(0.1, 10), method='bounded')
mle_lambda = res.x

print(f"Sample Mean: {np.mean(data):.4f}")
print(f"MLE Lambda: {mle_lambda:.4f}")

# Visualization
lam_range = np.linspace(2, 7, 100)
ll_values = [log_likelihood(l, data) for l in lam_range]

plt.figure(figsize=(10, 5))
plt.plot(lam_range, ll_values, label='Log-Likelihood', color='#2563eb', lw=2)
plt.axvline(mle_lambda, color='red', linestyle='--', label=f'MLE $\hat{{\lambda}}$ = {mle_lambda:.2f}')
plt.title('Log-Likelihood Surface for Poisson Parameter $\lambda$')
plt.xlabel('$\lambda$')
plt.ylabel('Log-Likelihood')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

Advanced Concepts: Efficiency

An estimator’s performance is often compared via Relative Efficiency: $Eff({\hat{\theta }}_1,{\hat{\theta }}_2)=\frac{Var({\hat{\theta }}_2)}{Var({\hat{\theta }}_1)}$ If the efficiency is $>1$, ${\hat{\theta }}_1$ is superior. As $n\to \infty$, the asymptotic efficiency of MLE is 1, meaning it is the best one can do in the limit.