Probability Spaces and Axiomatic Foundations

Probability isn’t about gut feelings or lucky guesses — it’s a precise language for describing uncertainty, built on a small set of rules that keep everything consistent. This chapter builds the foundation: we’ll see how to model random experiments, what counts as a sensible question about the outcome, and how to assign a numerical “chance” without any contradictions.

The Big Picture#

Imagine flipping a coin, rolling a die, or measuring tomorrow’s temperature. Each of these is a random experiment — we don’t know the result ahead of time, but we can list (or at least describe) all the things that might happen. Probability theory gives us a clean, three‑piece structure for handling such experiments: a set of all possible outcomes, a collection of “events” we can ask about, and a rule that attaches a number between 0 and 1 to every event. By starting from a few simple axioms, we can then figure out every other property of probability in a completely logical way. This chapter builds that structure from the ground up.

Outcomes and Sample Spaces#

Every experiment has an outcome — the single specific thing that actually happens when you carry it out.

Sample space (): The set of all possible outcomes of a random experiment.

Think of as the master list. If the experiment is a single coin toss, . For a six‑sided die, . If we measure the temperature in degrees Celsius, could be any real number, perhaps between and , so .

An outcome is often written as , so . Once the experiment is performed, exactly one occurs. The sample space can be finite (like a die), countably infinite (like the number of coin flips until the first Heads), or uncountable (like a physical measurement that can take any real value).

📝 Section Recap: The sample space is the complete set of possible outcomes. Everything else in probability is built on top of this set.

Events and Sigma‑Fields#

We rarely care only about a single raw outcome; we ask questions like “Did I roll an even number?” or “Is the temperature below freezing?” In the language of probability, such a question defines an event — a subset of .

Event: A subset that contains all the outcomes where the statement “ happens” is true.

In the die example, the event “roll an even number” is . The event “temperature below ” is the interval intersected with whatever we chose.

We would like to assign probabilities to events, but not every possible subset is necessarily a sensible event — especially when is uncountable. We need a well‑behaved collection of subsets. That collection is a sigma‑field (also called a sigma‑algebra).

Sigma‑field (): A collection of subsets of that satisfies three rules:

1. The whole space is in .
2. If a set is in , its complement is also in .
3. If you have countably many sets all in , then their union is also in .

The first two rules mean that always contains and the empty set (because ). The third rule, countable union closure, is the “sigma” part — it guarantees we can take unions of sequences of events without leaving the collection.

An event is simply any set that belongs to . If we can’t put a set into , then we don’t consider it a legitimate question for this particular model.

To see why sigma‑fields matter, suppose is a continuous interval. If we tried to let every subset be an event, we would run into deep mathematical problems when trying to define a probability measure that works for all subsets; that’s why we restrict ourselves to a carefully chosen . For countable sample spaces, we can usually take to be the power set (the collection of all subsets) without any trouble.

📝 Section Recap: Events are subsets of that belong to a sigma‑field . The sigma‑field tells us which sets are “measurable” — that is, which questions we are allowed to ask about the experiment.

The Probability Measure and Axioms#

Now we have the stage () and the list of admissible scenes (). The final piece is a rule that gives each event a number, its probability. This rule is a probability measure , and it must obey three axioms.

Probability measure (): A function that satisfies:

1. Non‑negativity: For every event , .
2. Normalization: .
3. Sigma‑additivity (countable additivity): If are pairwise disjoint events (no two share any outcome), then

These three axioms are the whole foundation. Everything else — complement rule, inclusion–exclusion, and so on — follows logically from them.

• Non‑negativity is obvious: a chance can’t be negative.
• Normalization means the probability that something happens is 1; the sample space is certain.
• Sigma‑additivity is the key: for a countable collection of mutually exclusive possibilities, the probability that one of them happens is just the sum of the individual probabilities.

When is a finite set and we take as the power set, sigma‑additivity collapses to finite additivity plus the fact that the probability of an empty union is zero. For an infinite sequence of disjoint events, we really need the full countable version to make sense of limits.

📝 Section Recap: A probability measure assigns a number between 0 and 1 to every event, with and the property that the chance of a countable union of non‑overlapping events is the sum of their chances.

First Consequences of the Axioms#

From the three axioms we can quickly prove a handful of simple, useful properties. Let’s go through them step by step.

Complement rule. Since and its complement are disjoint and their union is , so Thus the chance that something does not happen is one minus the chance that it does.

Finite additivity. Suppose we only have disjoint events . By defining for , sigma‑additivity gives because (proved by taking , for and using sigma‑additivity).

Monotonicity. If , we can write , where the two parts are disjoint. Then A larger set always has a probability at least as big as a smaller one inside it.

Subadditivity (Boole’s inequality). For any countable collection of events (not necessarily disjoint), We can prove this by “disjointifying” the sets: define , and for , The are disjoint and their union equals the original union, so because and monotonicity gives . This simple inequality tells us that the probability of a union is never more than the sum of the individual probabilities — a useful bound when exact calculations are hard.

Continuity from below and from above. These results let us handle limits of nested events.

• Continuity from below: If is an increasing sequence of events, then
• Continuity from above: If is a decreasing sequence, then Both follow from sigma‑additivity by writing the union or intersection in terms of disjoint differences. Continuity is essential for proving limit theorems later, but for now we can think of it as “probabilities behave nicely under limits of nested events.”

📝 Section Recap: From the three axioms we get a toolkit of rules: the complement rule, monotonicity, finite additivity, subadditivity, and continuity. These are the everyday workhorses of probability.

Constructing Sigma‑Fields#

When the sample space is simple — like a finite list — we can just take as the collection of all subsets (the power set). But for more complicated sets like , we can’t use the power set if we want to define a probability measure that works for all “nice” intervals. Instead, we start with a smaller collection of sets that we definitely want to be events, and then we “close” it under the sigma‑field operations to get the smallest sigma‑field that contains them.

Sigma‑field generated by a collection : Denoted , this is the intersection of all sigma‑fields that contain . It is the smallest sigma‑field that includes every set in .

If we have a collection of subsets of , we can always build because there is at least one sigma‑field containing (the power set), and the intersection of any family of sigma‑fields is again a sigma‑field. This construction is extremely useful: we specify the sets we absolutely need to measure, and the generated sigma‑field gives us the rest.

For example, if and we insist that the set must be an event, then will also contain , , , and all unions — so it’s the sigma‑field .

📝 Section Recap: The generated sigma‑field is the minimal collection of events that contains a starter set and satisfies the three sigma‑field rules. It’s the standard way to build a domain for a probability measure.

Borel Sets on the Real Line#

When we model quantities like temperature, weight, or time, the sample space is the real line (or an interval). The most natural events are intervals: “the temperature is between and ”, for instance. We want a sigma‑field that contains all intervals — and the smallest such sigma‑field is the Borel sigma‑field.

Borel sigma‑field ( or ): , the sigma‑field generated by the collection of all open intervals with .

It turns out that also contains closed intervals, half‑open intervals, singletons, and essentially every set we can describe without resorting to the axiom of choice. The Borel sets are exactly the “nice” subsets of for which we can consistently define a probability measure. We won’t prove the existence of such measures here, but we can assume that on there exist probability measures like the normal (Gaussian) distribution.

In practice, you can safely think of as “all the subsets of that you could ever build out of intervals using countable unions, intersections, and complements.” This is more than enough for any applied probability or statistics.

📝 Section Recap: The Borel sigma‑field is the smallest sigma‑field on that contains all open intervals. It gives us a rich collection of measurable sets suitable for continuous probability.

Discrete Probability Spaces#

If the sample space is finite or countably infinite, we can almost always take the sigma‑field to be the full power set . A probability measure on such a space is completely determined by the probabilities assigned to each single outcome. We call this a discrete probability measure.

Discrete probability measure: A probability measure on a countable space such that , where are non‑negative numbers with .

The function is often called a probability mass function (pmf). For a fair die, for each . For a biased coin, we might have , . For a countably infinite set, the sum of all still equals , which forces the probabilities to be small enough that the series converges.

Because we can use the power set, every subset is measurable, and sigma‑additivity reduces to the statement that the probability of a union of disjoint sets is the sum of their individual probability masses — exactly what the summation expression guarantees.

When you see a problem with dice, cards, or spinners, you’re usually working in a discrete space with a pmf, and the entire sigma‑field is the power set.

📝 Section Recap: On countable , we take as all subsets, and is defined by a probability mass function assigning a non‑negative weight to each outcome such that they sum to .

Product Spaces for Independent Experiments#

Many real‑world situations involve repeating the same random experiment several times — for example, flipping a fair coin twice. The natural sample space is the Cartesian product of the individual sample spaces. But what sigma‑field should we use? And how do we build a probability measure that respects independence?

Product sample space: If and are the sample spaces of two experiments, the combined sample space is , the set of all ordered pairs .

We need a sigma‑field on that allows us to talk about events concerning each experiment separately. A natural starting collection consists of measurable rectangles: sets of the form , where and (the sigma‑fields for the individual experiments). For instance, “first coin is Heads and second coin is Tails” is .

Product sigma‑field: , the smallest sigma‑field containing all measurable rectangles.

This construction ensures that sets built from rectangles — like “the sum of two dice is 7” — are measurable. It generalizes to any finite number of experiments, and with a bit more care, to infinite products as well.

Now, for independent experiments we want a probability measure on the product space that reflects the idea that outcomes of one experiment give no information about the other. The standard way is to define the probability of a rectangle as the product of the individual probabilities: Remarkably, this definition on rectangles uniquely extends to a probability measure on the whole product sigma‑field (a result from measure theory we won’t prove, but we’ll use it). The resulting product measure, often written , then satisfies that for any events and , the probability both occur is the product of their probabilities — the formal definition of independence.

For a coin flipped twice, with and , the product measure gives each of the four outcomes probability . The product sigma‑field contains all subsets, and everything lines up with our intuition.

📝 Section Recap: Product spaces allow us to model multiple experiments together. The product sigma‑field is generated by rectangles, and the product probability measure multiplies individual probabilities, giving us a rigorous way to handle independent trials.

Inclusion–Exclusion and Continuity of Probability#

We’ve already seen simple relations like the complement rule and subadditivity. Two more advanced — and extremely useful — tools are the inclusion–exclusion formula and the precise continuity properties.

Inclusion–exclusion formula (finite case). For any events , the probability of their union can be computed by alternating sums:

This formula corrects for over‑counting: we first add the single probabilities, then subtract the double‑intersections (which were counted twice), add back triple‑intersections, and so on. You can prove it by induction using the simpler two‑event version, Inclusion–exclusion is indispensable when events overlap and we need an exact union probability.

Continuity of probability (restated more formally). Because any probability measure is sigma‑additive, it enjoys the limit properties we sketched earlier:

• Continuity from below: If (meaning and ), then .
• Continuity from above: If (a decreasing sequence with intersection ), then .

These facts allow us to exchange limits and probabilities for monotone sequences, which is crucial when we study infinite trials or asymptotic results later.

📝 Section Recap: The inclusion–exclusion formula gives an exact method for finding the probability of a union of overlapping events. Continuity ensures that probabilities behave predictably as events approach a limit.

Summary#

We started with a blank slate and built the mathematical language of chance: what an outcome is, how to collect outcomes into events we can measure, and how to assign a number between 0 and 1 that obeys three simple rules. From those axioms we derived a whole toolkit — complement rule, monotonicity, subadditivity, continuity, and inclusion–exclusion — that makes probability reasoning both solid and easy to follow. Along the way, we learned how to create sigma‑fields from basic building blocks (leading to the useful Borel sets on the real line), how to handle countably many outcomes with a mass function, and how to combine independent experiments with product spaces. All of these ideas are the foundation on which the rest of probability theory rests.

Here is a quick reference for the central concepts: