Axioms

132. Axioms

132.1. Probability Axioms

Event: subset of the sample space
Probability $𝑃 (𝐴)$ is assigned to each event $𝐴$
After the experiment is performed, either the observed outcome lies in $𝐴$ (event $𝐴$ occurs) or it does not

132.1.1. Non-Negativity Axiom

Probability of any event is always non-negative

𝑃 (𝐴) \geq 0

132.1.2. Normalization Axiom

The probability of the sample space is always 1

𝑃 (Ω) = 1

132.1.3. Finite Additivity Axiom

If two events are disjoint (mutually exclusive),

𝐴 \cap 𝐵 = \emptyset

then

𝑃 (𝐴 \cup 𝐵) = 𝑃 (𝐴) + 𝑃 (𝐵)

Consequences of the Axioms:

132.1.4. $0 \leq 𝑃 (𝐴) \leq 1$

Since

𝐴 \cup 𝐴^{𝑐} = Ω, 𝐴 \cap 𝐴^{𝑐} = \emptyset

we have

1 = 𝑃 (Ω) = 𝑃 (𝐴) + 𝑃 (𝐴^{𝑐})

Thus,

𝑃 (𝐴) = 1 - 𝑃 (𝐴^{𝑐}) \overset{(𝑎)}{\leq} 1

132.1.5. $𝑃 (\emptyset) = 0$

\begin{aligned} 1 & = 𝑃 (Ω) \\ = 𝑃 (Ω) + 𝑃 (Ω^{𝑐}) \\ = 1 + 𝑃 (\emptyset) \Rightarrow 𝑃 (\emptyset) = 0 \end{aligned}

132.1.6. $𝑃 (𝐴) + 𝑃 (𝐴^{𝑐}) = 1$

Follows immediately from additivity applied to $𝐴$ and $𝐴^{𝑐}$ .

132.1.7. Probability of Finite Unions of Disjoint Events

For three mutually disjoint sets:

𝑃 (𝐴 \cup 𝐵 \cup 𝐶) = 𝑃 (𝐴) + 𝑃 (𝐵) + 𝑃 (𝐶)

More generally, if:

𝐴_{𝑖} \cap 𝐴_{𝑗} = \emptyset for 𝑖 \neq 𝑗

then:

From (Finite) Additivity, we have:

𝑃 (𝐴 \cup 𝐵) = 𝑃 (𝐴) + 𝑃 (𝐵)

So,

\begin{aligned} 𝑃 (𝐴 \cup 𝐵 \cup 𝐶) & = 𝑃 ((𝐴 \cup 𝐵) \cup 𝐶) \\ = 𝑃 (𝐴 \cup 𝐵) + 𝑃 (𝐶) \\ = 𝑃 (𝐴) + 𝑃 (𝐵) + 𝑃 (𝐶) \end{aligned}

More generally, by induction:

𝑃 (⋃_{𝑖 = 1}^{𝑛} 𝐴_{𝑖}) = \sum_{𝑖 = 1}^{𝑛} 𝑃 (𝐴_{𝑖}) 𝐴_{𝑖} \cap 𝐴_{𝑗} = \emptyset, for 𝑖 \neq 𝑗

132.1.8. If $𝐴 \subset 𝐵$ then $𝑃 (𝐴) \leq 𝑃 (𝐵)$

\begin{matrix} 𝐵 = 𝐴 \cup (𝐵 \cap 𝐴^{𝑐}) \\ 𝑃 (𝐵) = 𝑃 (𝐴) + 𝑃 (𝐵 \cap 𝐴^{𝑐}) \geq 𝑃 (𝐴) \end{matrix}

132.1.9. $𝑃 (𝐴 \cup 𝐵) = 𝑃 (𝐴) + 𝑃 (𝐵) = 𝑃 (𝐴 \cap 𝐵)$

𝑃 (𝐴 \cup 𝐵) = 𝑃 (𝐴 \cap 𝐵^{𝑐}) + 𝑃 (𝐴 \cap 𝐵) + 𝑃 (𝐵 \cap 𝐴^{𝑐})

\begin{aligned} 𝑃 (𝐴) + 𝑃 (𝐵) - 𝑃 (𝐴 \cap 𝐵) & = (𝑃 (𝐴 \cap 𝐵^{𝑐}) + 𝑃 (𝐴 \cap 𝐵)) + (𝑃 (𝐴 \cap 𝐵) + 𝑃 (𝐵 \cap 𝐴^{𝑐})) - 𝑃 (𝐴 \cap 𝐵) \\ = 𝑃 (𝐴 \cap 𝐵^{𝑐}) + 𝑃 (𝐴 \cap 𝐵) + 𝑃 (𝐵 \cap 𝐴^{𝑐})) \end{aligned}

132.1.10. Union Bound

𝑃 (𝐴 \cup 𝐵) \leq 𝑃 (𝐴) + 𝑃 (𝐵)

Since $𝑃 (𝐴 \cap 𝐵) \geq 0$

132.1.11. Union of more than two overlapping (non disjoint) sets

𝑃 (𝐴 \cup 𝐵 \cup 𝐶) = 𝑃 (𝐴) + 𝑃 (𝐴^{𝑐} \cap 𝐵) + 𝑃 (𝐴^{𝑐} \cap 𝐵^{𝑐} \cap 𝐶)

\begin{aligned} 𝐴 \cup 𝐵 \cup 𝐶 & = 𝐴 \cup 𝐵 \cap 𝐴^{𝑐} \cup 𝐶 \cap 𝐴^{𝑐} \cap 𝐵^{𝑐} \\ 𝑃 (𝐴 \cup 𝐵 \cup 𝐶) & = 𝑃 (𝐴) + 𝑃 (𝐵 \cap 𝐴^{𝑐}) + 𝑃 (𝐶 \cap 𝐴^{𝑐} \cap 𝐵^{𝑐}) \end{aligned}

132.2. Discrete Uniform Law

Assume $Ω$ consists of $𝑛$ equally likely elements
Assume $𝐴$ consists of $𝑘$ elements

𝑃 (𝐴) = 𝑘 \frac{1}{𝑛}

132.3. Probability Calculations Steps

Specify a sample space
Specify a probability law
Identify an event of interest
Calculate

Discrete but infinite sample space

Example

Number of coin tosses until we observe a heads toss

Sample Space

Ω = {1, 2, \dots, \infty}

We are given $𝑃 (𝑛) = \frac{1}{2^{𝑛}}, 𝑛 = 1, 2, \dots$

\sum_{𝑛 = 1}^{\infty} \frac{1}{2^{𝑛}} = \frac{1}{2} \sum_{𝑛 = 0}^{\infty} \frac{1}{2^{𝑛}} = \frac{1}{2} \cdot \frac{1}{1 - (\frac{1}{2})} = 1

\begin{aligned} 𝑃 (outcome is even) & = 𝑃 ({2, 4, \dots}) \\ = 𝑃 ({2} \cup {4} \cup \dots) \\ = 𝑃 (2) + 𝑃 (4) + \dots \\ = \frac{1}{2^{2}} + \frac{1}{2^{4}} + \dots \\ \frac{1}{4} (1 + \frac{1}{4} + \frac{1}{4^{2}} + \dots) \\ = \frac{1}{4} \cdot \frac{1}{1 - \frac{1}{4}} \\ = \frac{1}{3} \end{aligned}

132.3.1. Countable Additivity Axiom

If $𝑎_{1}, 𝐴_{2}, 𝐴_{3} \dots$ is an infinite sequence of disjoint events, then $𝑃 (𝐴_{1} \cup 𝐴_{2} \cup 𝐴_{3} \dots) = 𝑃 (𝐴_{1}) + 𝑃 (𝐴_{2}) + 𝑃 (𝐴_{3}) + \dots$

𝑃 (⋃_{𝑖 = 1}^{\infty} 𝐴_{𝑖}) = \sum_{𝑖 = 1}^{\infty} 𝑃 (𝐴_{𝑖}) 𝐴_{𝑖} \cap 𝐴_{𝑗} = \emptyset, for 𝑖 \neq 𝑗

Additivity holds only for “countable” sequences of events
- The unit square (or real line) if not countable