Axiomatic Probability Theory

Probability theory is the mathematical framework for quantifying uncertainty. While intuitive and historical approaches (like frequency or subjective belief) exist, modern probability is built upon the rigorous foundations of measure theory, established by Andrey Kolmogorov in 1933.

1. The Kolmogorov Axioms

A probability model is defined by a probability space $(\Omega ,\mathcal{F},P)$, where:

1. Sample Space ($\Omega$): The set of all possible outcomes of a random experiment.
2. $\sigma$-algebra ($\mathcal{F}$): A collection of subsets of $\Omega$ (called events) that is closed under complements and countable unions. Formally:
   • $\varnothing \in \mathcal{F}$.
   • If $A\in \mathcal{F}$, then $A^c\in \mathcal{F}$.
   • If $A_1,A_2,\cdots \in \mathcal{F}$, then ${\bigcup }_{i=1}^{\infty }{A}_i\in \mathcal{F}$.
3. Probability Measure ($P$): A function $P:\mathcal{F}\to [0,1]$ satisfying:
   • Non-negativity: $P(A)\ge 0$ for all $A\in \mathcal{F}$.
   • Normalization: $P(\Omega )=1$.
   • Countable Additivity: For any sequence of disjoint events $A_1,A_2,\dots$, $P\left({\bigcup }_{i=1}^{\infty }{A}_i\right)={\sum }_{i=1}^{\infty }P(A_i)$.

From these axioms, we derive the fundamental properties of probability, such as $P(A^c)=1-P(A)$ and the inclusion-exclusion principle.

2. Conditional Probability and Bayes’ Theorem

For two events $A,B\in \mathcal{F}$ with $P(B)>0$, the conditional probability of $A$ given $B$ is defined as: $P(A\mid B)=\frac{P(A\cap B)}{P(B)}$

The Law of Total Probability

If $\{B_i{\}}_{i=1}^n$ is a partition of $\Omega$, then for any event $A$: $P(A)={\sum }_{i=1}^{n}P(A\mid B_i)P(B_i)$

Bayes’ Theorem

Bayes’ theorem allows us to invert conditional probabilities, forming the basis of Bayesian inference: $P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{{\sum }_{i=1}^{n}P(A\mid B_i)P(B_i)}$

3. Random Variables

A random variable $X$ is not a variable in the algebraic sense, but a measurable function $X:\Omega \to \mathbb{R}$. This means that for every Borel set $B\in \mathcal{B}(\mathbb{R})$, the pre-image $X^{-1}(B)=\{\omega \in \Omega :X(\omega )\in B\}$ is an element of $\mathcal{F}$.

Discrete vs. Continuous Random Variables

• Discrete: $X$ takes values in a countable set. Its distribution is described by a Probability Mass Function (PMF) $p_X(x)=P(X=x)$.
• Continuous: $X$ takes values in an uncountable set (usually $\mathbb{R}$). Its distribution is described by a Probability Density Function (PDF) $f_X(x)$ such that: $P(a\le X\le b)={\int }_a^b{f}_X(x)dx$ The Cumulative Distribution Function (CDF) is defined for all random variables as $F_X(x)=P(X\le x)$.

4. Expectation and Moments

The expected value $E[X]$ is the “center of mass” of the distribution. In measure-theoretic terms, it is the Lebesgue integral: $E[X]={\int }_{\Omega }XdP$.

• For discrete $X$: $E[X]=\sum x{p}_X(x)$
• For continuous $X$: $E[X]={\int }_{-\infty }^{\infty }x{f}_X(x)dx$

Properties of Expectation

• Linearity: $E[aX+bY]=aE[X]+bE[Y]$, regardless of independence.
• Variance: $\text{Var}(X)=E[(X-E[X]{)}^2]=E[X^2]-(E[X]{)}^2$.

Moment Generating Functions (MGF)

The MGF of a random variable $X$ is defined as: $M_X(t)=E[e^{tX}]$ If $M_X(t)$ exists in a neighborhood around $t=0$, it uniquely determines the distribution. Moments can be found by differentiating: $E[X^n]=M_X^{(n)}(0)$.

The Characteristic Function ${\varphi }_X(t)=E[e^{itX}]$ always exists for any distribution and is the Fourier transform of the density function.

5. Probability Inequalities

Inequalities provide upper bounds on the probability of “tail events.”

1. Markov’s Inequality: For a non-negative random variable $X$ and $a>0$: $P(X\ge a)\le \frac{E[X]}{a}$
2. Chebyshev’s Inequality: For any $X$ with mean $\mu$ and variance ${\sigma }^2$: $P(|X-\mu |\ge k\sigma )\le \frac{1}{k^2}$
3. Jensen’s Inequality: For a convex function $\varphi$: $\varphi (E[X])\le E[\varphi (X)]$

6. Limit Theorems

Limit theorems describe the behavior of the sum of independent and identically distributed (i.i.d.) random variables.

Law of Large Numbers (LLN)

Let $X_1,X_2,\dots$ be i.i.d. random variables with $E[X_i]=\mu$. Let ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$.

• Weak Law (WLLN): ${\bar{X}}_n\stackrel{P}{\to }\mu$ (Convergence in probability).
• Strong Law (SLLN): ${\bar{X}}_n\stackrel{a.s.}{\to }\mu$ (Almost sure convergence).

Central Limit Theorem (CLT)

Let $X_1,X_2,\dots$ be i.i.d. with mean $\mu$ and finite variance ${\sigma }^2$. As $n\to \infty$: $\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}\stackrel{d}{\to }\mathcal{N}(0,1)$ This explains why the Normal distribution appears everywhere in nature: it represents the aggregate effect of many independent small fluctuations.

Python Simulation: The Central Limit Theorem

The following script simulates the CLT by summing variables from a non-normal distribution (Uniform) and showing the resulting distribution of the mean.

import numpy as np
import matplotlib.pyplot as plt

def simulate_clt(sample_size, num_simulations):
    # Generating samples from a Uniform(0, 1) distribution
    # Mean = 0.5, Variance = 1/12
    means = []
    for _ in range(num_simulations):
        data = np.random.uniform(0, 1, sample_size)
        means.append(np.mean(data))
    
    plt.figure(figsize=(10, 6))
    plt.hist(means, bins=50, density=True, alpha=0.7, color='steelblue')
    
    # Overlay the theoretical Normal distribution
    mu = 0.5
    sigma = np.sqrt(1/12) / np.sqrt(sample_size)
    x = np.linspace(min(means), max(means), 100)
    p = (1 / (sigma * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - mu) / sigma)**2)
    plt.plot(x, p, 'r', linewidth=2)
    
    plt.title(f"CLT Simulation: n={sample_size}")
    plt.xlabel("Sample Mean")
    plt.ylabel("Density")
    plt.show()

simulate_clt(sample_size=30, num_simulations=10000)