Probability Foundations and Conditional Expectation

Probability is the language we use to talk about uncertainty. In this chapter we build the precise grammar of that language — sigma-fields, random variables, and conditional expectation — so we can later describe how randomness evolves over time. Don’t worry if some of these ideas feel abstract; we’ll walk through them one by one, and by the end you’ll see why the tower property of conditional expectation is one of the most useful shortcuts in all of stochastic calculus.

The Big Picture#

Everything we do from here on rests on a handful of core probability concepts. We need a precise way to talk about “information” — what we know and when we know it. Sigma-fields turn out to be the perfect tool for that. Once we understand them, we can revisit random variables as functions that translate outcomes into numbers, and we can define conditional expectation not just as a single number, but as a random variable that depends on the information we have. The tower property then gives us a clean rule for averaging in stages, which will become essential when we study martingales and Ito integrals.

Sigma-fields and Measurability#

Imagine you toss a fair coin twice. Before you toss, all four outcomes {HH, HT, TH, TT} are possible. After the first toss, you know whether it landed heads or tails, but you still don’t know the second result. Your information has changed — the set of events you can decide has grown. Sigma-fields are the mathematical object that captures exactly which events are “knowable” at a given moment.

Start with a sample space , the set of all possible outcomes. A sigma-field (or sigma-algebra) is a collection of subsets of that satisfies three rules:

• The whole space is in .
• If a set is in , its complement is also in .
• If is a countable sequence of sets in , their union is in .

These conditions simply say that a sigma-field is closed under the logical operations of “everything,” “not,” and “countable or.” Any set in the sigma-field is called a measurable set or an event.

Sigma-field: A family of subsets of containing , closed under complements and countable unions.

The simplest sigma-field is . It represents knowing nothing except that something happens. At the other extreme, the power set — the collection of all subsets of — represents complete knowledge. Between these extremes lie sigma-fields that capture partial information. In the two-coin-toss example, after the first toss we can decide events like “the first toss is heads” , but we cannot yet decide events that depend on both tosses. The sigma-field that corresponds to this information contains exactly those sets that only care about the first toss.

Now suppose we have a probability measure on — a function that assigns a number between and to each event, with and the property that the probability of a union of disjoint events is the sum of their individual probabilities (countable additivity). The triple is our probability space. The sigma-field tells us which questions we are allowed to ask; gives us the answers.

A closely related idea is a sub-sigma-field . If contains fewer sets than , it represents coarser information. Thinking of sigma-fields as information sets will pay off enormously when we define conditional expectation.

📝 Section Recap: A sigma-field is a collection of events closed under the basic operations of logic, and it models the information available in a probability experiment. Sub-sigma-fields represent partial knowledge.

Random Variables and Distributions#

A random variable is often introduced as something that takes on numerical values randomly. More precisely, a real-valued random variable on a probability space is a function that is measurable with respect to . This means: for every real number , the set of outcomes where is an event in . In symbols,

Measurability ensures that we can assign a probability to all questions of the form “does fall in this interval?” because those questions correspond to sets in our sigma-field. Without this condition, we could not consistently talk about .

Random variable: A measurable function from a probability space to the real numbers.

A random variable’s behavior is completely described by its distribution, which tells us the probability that lands in any Borel set (a set built from intervals using countable operations). The cumulative distribution function (CDF) is often enough to determine the distribution.

The expectation is the probability-weighted average of . For a simple random variable that takes finitely many values, it is a sum of value times probability. For a general random variable, it is defined by an integral with respect to : . We will only work with integrable random variables, those for which ; this guarantees the expectation exists and is finite.

Two random variables and can be equal in different senses. They are almost surely equal, written , if . They have the same distribution if for all . Equal almost surely implies equal in distribution, but not conversely — think of two independent fair coin flips, each modeled by a random variable that is for heads and for tails; they have the same distribution but are not equal on the same outcome.

Why do we insist on measurability? Because the sigma-field tells us what is “knowable.” If is -measurable, then knowing the true outcome determines . If we only have access to a smaller sigma-field , a -measurable might not be -measurable — we might not be able to compute it from the limited information in . Conditional expectation will later let us form a “best guess” of using only the information in .

📝 Section Recap: A random variable is a measurable function from the sample space to numbers; its distribution captures the probabilities of its values. Measurability with respect to a sigma-field ensures that all probability statements about the variable make sense within our model.

Conditional Expectation Given a Sigma-field#

Most students first meet conditional expectation as a number: “what is the expected value of given that ?” This works well for discrete and continuous variables, but it cannot handle the rich information structures that appear in stochastic processes. We need a more general definition that conditions on an entire sigma-field, not just a single event.

Let be an integrable random variable on , and let be a sub-sigma-field of (so ). The conditional expectation of given is a random variable, denoted , that satisfies two properties:

1. -measurability: is measurable with respect to the smaller sigma-field .
2. Partial averaging: For every set ,

In plain words: is the “best -measurable approximation” to , in the sense that it matches on all averages taken over events in . If represents coarser information, then is what we would guess for knowing only that information.

Any two versions of that satisfy (1) and (2) are almost surely equal, so we speak of “the” conditional expectation.

Notice how this definition generalizes the elementary case. If is the sigma-field generated by a simple random variable taking finitely many values, then the events partition . The partial averaging condition forces to be constant on each , and that constant equals the elementary .

Conditional expectation (given ): A random variable that uses only information in and gives the same integral as over every event in .

Key properties follow directly from the definition:

• Linearity: almost surely.
• Monotonicity: If almost surely, then almost surely.
• Pulling out what is known: If is -measurable and bounded, then almost surely. Because is already determined by the information in , it behaves like a constant inside the conditional expectation.
• Independence: If is independent of (every event in is independent of the behavior of ), then almost surely. Partial information that tells us nothing about gives no better guess than the overall average.
• Law of total expectation (special case): Taking in (2) gives . The average of the conditional guess is the overall average.

Think of a weather forecaster. They receive a large information set (satellite images, pressure readings, historical data). They need to produce a forecast accessible to the public — that forecast only uses a coarser information set (say, whether it will rain or not in each region). The forecast is a -measurable random variable, and if it is a fair forecast, over many days with the same satellite patterns, the average actual rain amount equals the forecast average. That’s exactly condition (2).

📝 Section Recap: Conditional expectation given a sigma-field is a random variable that refines a coarser information set by averaging out the unknowable details, while preserving averages on all events in that set.

The Tower Property#

The most powerful computational property of conditional expectation is the tower property (also called the law of iterated expectations, but generalized to sigma-fields). It tells you how to condition in stages.

Suppose we have a nested sequence of sigma-fields:

is coarser than , which is coarser than the full information . Then for any integrable random variable ,

Why is it called the tower property? Because if you stack the conditional expectations, the “inner” ones collapse: you can average first with respect to and then with respect to , and you get the same result as if you had averaged directly with respect to .

A special case occurs when , the trivial sigma-field. Then , the ordinary number. The property becomes

which is the familiar law of total expectation. But the tower property is far more general because can be any coarser sigma-field, not just the trivial one.

Here is a short proof. Let . For any event , since , is also in . By the definition of ,

By the definition of ,

Now apply the definition of : it is the -measurable random variable that satisfies for all . Combining the equalities,

Both and are -measurable, and they have the same integral over every . Therefore they are equal almost surely. That proves the tower property.

Let’s illustrate with a concrete example. Suppose you first roll a fair die, call the outcome . Then, given , you flip a fair coin times and count the number of heads . What is the expected number of heads if you only know ? That’s , where is the sigma-field generated by . Clearly , because each flip has probability of heads. Now suppose you want the unconditional expectation . The tower property with gives . Since is uniform on , , so . This two-step averaging is much simpler than working out the entire distribution of .

The tower property becomes essential when we study martingales and Ito integrals. It tells us that a martingale’s conditional expectation on a smaller sigma-field recovers its earlier value, and it justifies many “condition then integrate” simplifications in stochastic calculus. Whenever you can break a complex expectation into a chain of simpler conditional expectations, the tower property is your tool.

📝 Section Recap: The tower property lets you condition in steps: the conditional expectation of a conditional expectation, given a coarser sigma-field, collapses to the conditional expectation given that coarser field. It is the general form of the law of total expectation and a cornerstone of stochastic analysis.

Summary#

We started with the idea that a sigma-field is a formal way to describe what information is available, and random variables are the measurable quantities we observe. Then we moved to conditional expectation, which gives us a way to form a best guess of a random variable using only partial information. The tower property, which says that conditioning in stages collapses down to a single conditioning, is the glue that ties all these concepts together. These foundations will reappear over and over as we build stochastic processes and eventually stochastic integrals — every step will rely on the language and rules you have just learned.