Distributions and Density Functions

Probability theory provides the rigorous framework for modeling uncertainty in physical, social, and information systems. At the heart of this framework lie random variables and the functions that describe their probabilistic behavior. This lesson details the characterization of random variables, the catalog of fundamental distributions, and the calculus of transformations.

1. Probability Density Functions (PDF) and Cumulative Distribution Functions (CDF)

A Random Variable $X$ is a measurable function $X:\Omega \to \mathbb{R}$ mapping the sample space to the real line. The behavior of $X$ is fully characterized by its Cumulative Distribution Function (CDF):

$F_X(x)=P(X\le x)$

The CDF is non-decreasing, right-continuous, and satisfies ${\lim }_{x\to -\infty }{F}_X(x)=0$ and ${\lim }_{x\to \infty }{F}_X(x)=1$. For a Continuous Random Variable, the Probability Density Function (PDF) $f_X(x)$ exists such that:

$F_X(x)={\int }_{-\infty }^x{f}_X(t)dt⟺f_X(x)=\frac{d}{dx}{F}_X(x)$

The fundamental properties of a PDF include:

1. $f_X(x)\ge 0$ (Non-negativity)
2. ${\int }_{-\infty }^{\infty }{f}_X(x)dx=1$ (Normalization)

The probability that $X$ falls within an interval $[a,b]$ is given by: $P(a\le X\le b)=F_X(b)-F_X(a)={\int }_a^b{f}_X(x)dx$

2. Catalog of Distributions

2.1 Discrete Distributions

Discrete distributions model variables that take values in a countable set.

Binomial Distribution ($X\sim \text{Bin}(n,p)$): Models the number of successes in $n$ independent Bernoulli trials with success probability $p$. $P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k},k\in \{0,1,\dots ,n\}$ Expected Value: $E[X]=np$; Variance: $\text{Var}(X)=np(1-p)$.

Poisson Distribution ($X\sim \text{Pois}(\lambda )$): Models the number of arrivals in a fixed interval given a constant average rate $\lambda$. $P(X=k)=\frac{{\lambda }^k{e}^{-\lambda }}{k!},k\in \{0,1,2,\dots \}$ The Poisson distribution is often used as a limit of the Binomial distribution as $n\to \infty$ and $p\to 0$ with $np=\lambda$.

2.2 Continuous Distributions

Gaussian (Normal) Distribution ($X\sim \mathcal{N}(\mu ,{\sigma }^2)$): The most significant distribution in statistics due to the Central Limit Theorem (CLT), which asserts that the sum of independent and identically distributed (i.i.d.) random variables converges to a normal distribution. $f(x)=\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$

Exponential Distribution ($X\sim \text{Exp}(\lambda )$): Models the time between events in a Poisson process. It is the only continuous distribution with the memoryless property: $P(X>s+t\mid X>s)=P(X>t)$ PDF: $f(x)=\lambda e^{-\lambda x}$ for $x\ge 0$.

Gamma Distribution: Generalizes the exponential distribution. $X\sim \text{Gamma}(\alpha ,\beta )$ models the time until the $\alpha$-th event. $f(x)=\frac{{\beta }^{\alpha }}{\Gamma (\alpha )}{x}^{\alpha -1}{e}^{-\beta x},\text{where }\Gamma (\alpha )={\int }_0^{\infty }{t}^{\alpha -1}{e}^{-t}dt$

Beta Distribution: Defined on $[0,1]$, making it ideal for modeling probabilities or proportions. $f(x)=\frac{1}{B(\alpha ,\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1},B(\alpha ,\beta )=\frac{\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha +\beta )}$

3. Multivariate Distributions

In many applications, we track multiple random variables simultaneously. The Joint PDF $f_{X,Y}(x,y)$ describes their simultaneous behavior.

3.1 Marginal and Conditional Densities

The Marginal Density of $X$ is obtained by integrating out $Y$: $f_X(x)={\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dy$

The Conditional Density of $Y$ given $X=x$ is: $f_{Y|X}(y|x)=\frac{f_{X,Y}(x,y)}{f_X(x)}$

3.2 Independence

Random variables $X$ and $Y$ are independent if and only if their joint density factors into their marginals: $f_{X,Y}(x,y)=f_X(x)f_Y(y)$ This implies that $f_{Y|X}(y|x)=f_Y(y)$; knowing $X$ provides no information about $Y$.

4. Covariance and Correlation

Covariance measures the joint variability of two variables: $\text{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y]$

For a random vector $\mathbf{X}=[X_1,X_2,\dots ,X_n{]}^T$, we define the Covariance Matrix $\Sigma$: $\Sigma =E[(\mathbf{X}-E[\mathbf{X}])(\mathbf{X}-E[\mathbf{X}]{)}^T]$ Where ${\Sigma }_{ij}=\text{Cov}(X_i,X_j)$. $\Sigma$ is always symmetric and positive semi-definite. If $X_i$ are independent, $\Sigma$ is diagonal.

5. Transformation of Random Variables

When we define a new variable $Y=g(X)$, we must determine its density $f_Y(y)$.

5.1 Univariate Transformation

If $g$ is strictly monotonic and differentiable: $f_Y(y)=f_X(g^{-1}(y))\left|\frac{d}{dy}{g}^{-1}(y)\right|$

5.2 Multivariate Transformation and the Jacobian

For a vector transformation $\mathbf{Y}=\mathbf{g}(\mathbf{X})$, let $\mathbf{h}={\mathbf{g}}^{-1}$ be the inverse mapping. The joint density follows: $f_{\mathbf{Y}}(\mathbf{y})=f_{\mathbf{X}}(\mathbf{h}(\mathbf{y}))\left|\det (J_{\mathbf{h}}(\mathbf{y}))\right|$ where $J_{\mathbf{h}}$ is the Jacobian Matrix of the inverse transformation: $J_{\mathbf{h}}=\frac{\partial (x_1,\dots ,x_n)}{\partial (y_1,\dots ,y_n)}$

6. Sampling Distributions

Statistical inference relies on the distributions of sample statistics.

1. Chi-squared Distribution (${\chi }_k^2$): If $Z_1,\dots ,Z_k$ are independent standard normal variables, then $\sum Z_i^2\sim {\chi }_k^2$.
2. Student’s t-Distribution ($t_k$): Arises when estimating the mean of a normally distributed population with unknown variance. $T=\frac{Z}{\sqrt{V/k}}$ where $Z\sim \mathcal{N}(0,1)$ and $V\sim {\chi }_k^2$.
3. F-Distribution ($F_{d_1,d_2}$): The ratio of two scaled chi-squared variables; fundamental for comparing variances and ANOVA.

7. Numerical Analysis with SciPy

The following Python snippet demonstrates the verification of the transformation $Y=X^2$ using the scipy.stats library. If $X$ is a standard normal variable, its square follows a Chi-squared distribution with 1 degree of freedom.

import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# Generate standard normal samples
n_samples = 100000
x_samples = np.random.normal(loc=0, scale=1, size=n_samples)

# Transform: Y = X^2
y_samples = x_samples**2

# Analytical Chi-square PDF (k=1)
y_range = np.linspace(0.01, 6, 1000)
pdf_analytical = stats.chi2.pdf(y_range, df=1)

# Plotting
plt.figure(figsize=(12, 6))
plt.hist(y_samples, bins=100, density=True, alpha=0.5, label='Empirical Histogram (X^2)')
plt.plot(y_range, pdf_analytical, 'r-', lw=2, label='Analytical Chi2(df=1)')
plt.title("Verification of Variable Transformation: $X \sim \mathcal{N}(0,1) \implies X^2 \sim \chi^2_1$")
plt.xlabel("Value")
plt.ylabel("Density")
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

# Demonstrate the Exponential Memoryless Property
lambda_param = 0.5
t_survive = 2.0  # Already survived 2 units
t_additional = 1.0 # Probability of surviving 1 more unit

# P(X > t_survive + t_additional | X > t_survive)
p_cond = (1 - stats.expon.cdf(t_survive + t_additional, scale=1/lambda_param)) / \
         (1 - stats.expon.cdf(t_survive, scale=1/lambda_param))
# P(X > t_additional)
p_orig = 1 - stats.expon.cdf(t_additional, scale=1/lambda_param)

print(f"Conditional Prob: {p_cond:.5f}")
print(f"Unconditional Prob: {p_orig:.5f}")
print(f"Difference: {abs(p_cond - p_orig):.10f}")