Conditional Probability and Independence

If you have ever changed your guess after seeing new information—like rethinking the chance of rain when you spot dark clouds—you already have an intuitive feel for conditional probability. In this chapter we make that idea precise. Then we learn how to break complex problems into simpler conditional pieces, how to reverse a conditional question with Bayes’ formula, and finally what it really means for events and random variables to be independent.

The Big Picture#

Probability is all about reasoning with uncertainty. But often we don’t see all possibilities at once—we get a clue, a piece of data, that narrows the field. This chapter answers one big question: how do we update our probabilities when we learn that something has happened? From that question we get the Law of Total Probability and Bayes’ formula—two very practical tools for data analysis, medicine, and everyday decisions. Then we look at the opposite idea: when does knowing one thing give you no information about another? That’s independence, and it’s the foundation of many probability models.

Conditional Probability#

Imagine you roll a fair six‑sided die. The probability of rolling a 4 is . But suppose a friend peeks and tells you, “the outcome is even.” Now the only possible results are , each equally likely, so the probability of a 4 jumps to . You have conditioned on the event “even.”

We write this situation with a simple formula:

Conditional probability: The conditional probability of event given event , written , is

provided .

The denominator acts as a scaling factor: we shrink the sample space down to only those outcomes where happens, then measure how much of is also in .

It helps to picture a Venn diagram. Think of a rectangle as the whole sample space. The event is a blob inside it. Once we know occurred, we restrict our world to that blob. The probability of given is the fraction of the blob that overlaps with .

A short example. Pick a card at random from a standard 52‑card deck. Let be “the card is a king” and be “the card is a face card” (jack, queen, king). There are face cards, kings, and kings are face cards. So and . The conditional probability is

Makes sense: among face cards, one third are kings.

Note that once we condition on , the conditional probability behaves just like an ordinary probability—it follows all the same rules.

Sometimes we flip the condition around: is not the same as . Confusing them is a classic mistake. We will soon see a tool to relate them.

📝 Section Recap: Conditional probability updates our probability of by restricting the sample space to and rescaling; it is the ratio .

The Law of Total Probability#

Real‑world problems often come in layers. You want the probability of some event , but its chance depends on which of several mutually exclusive scenarios occurs. The Law of Total Probability lets us combine the conditional pieces.

We start with a partition of the sample space: a set of events that are

mutually exclusive: for ,
exhaustive: their union is the whole sample space, .

Think of it as chopping the whole universe into non‑overlapping pieces, like slicing a pie. Then for any event ,

Law of Total Probability:

In words: the overall probability of is the weighted average of its conditional probabilities across the partition, weighted by the probabilities of the partition pieces.

Why does this hold? Because can happen together with exactly one of the (since they cover everything and don’t overlap), so

Example: A test with two paths. Suppose a factory has two machines that produce light bulbs. Machine 1 makes of the bulbs, and of its bulbs are defective. Machine 2 makes of the bulbs, and of its bulbs are defective. If a bulb is selected at random from a day’s total output, what is the probability it is defective?

Let be “defective,” be “made by machine 1,” be “made by machine 2.” Then and form a partition. We know , , , . The law gives

So a randomly chosen bulb has a chance of being defective.

The Law of Total Probability is a “divide and conquer” strategy: split a hard problem into easier sub‑problems and combine the answers. It is especially useful when is far easier to estimate than directly.

📝 Section Recap: The Law of Total Probability expresses as a sum of conditional probabilities weighted by , where the partition the sample space. It turns a tough probability into a manageable weighted average.

Bayes’ Formula#

Sometimes we know but we really want the reverse: . In the factory example, if you find a defective bulb, which machine was it more likely to have come from? That is a job for Bayes’ formula.

Starting from the definition of conditional probability,

Now replace the denominator with the Law of Total Probability, using a partition that includes the we care about. For a particular ,

Bayes’ formula:

The numerator is the ‑th piece of the total probability sum; the denominator is the whole sum. So Bayes’ formula simply says: take each possible cause , multiply its prior probability by how likely it makes the evidence , and then normalize so that all the probabilities sum to .

This is a mathematical version of “update your prior beliefs with evidence.” We start with a prior probability —our belief before observing . We collect evidence through the likelihood . The denominator, often called the normalising constant, ensures everything adds to one. The result is the posterior probability , our updated belief.

Example: disease testing. Suppose a rare disease affects in people (). A test for the disease is accurate: it yields a positive result for of those with the disease () and a negative result for of those without it (, so ). If a person tests positive, what is the probability they actually have the disease?

Let the partition be (disease) and (healthy). We want . Bayes’ formula gives

Plug in numbers:

Even with a positive test, the probability of actually having the disease is only about —a surprise to most people! The disease is so rare that almost all positive results are false positives. Bayes’ formula quantifies that intuition.

Bayes’ formula is the basis of many methods: Bayesian statistics, machine‑learning classifiers, spam filters, and medical diagnosis. It is simply a logical way to flip a conditional probability.

📝 Section Recap: Bayes’ formula computes the reverse conditional probability by multiplying the prior by the likelihood and normalising by the total probability. It is the optimal recipe for updating beliefs when new evidence arrives.

Independence of Events#

Conditional probability tells us how events are linked. The opposite extreme is when knowing one event tells you absolutely nothing about another—they are independent.

Intuitively, and are independent if the occurrence of does not change the probability of , i.e., . Using the conditional formula, this becomes (as long as ). We take this product rule as the formal definition because it also works when and it is symmetric.

Independent events: Two events and are independent if

Example: coin and die. Toss a fair coin and roll a fair six‑sided die. The combined experiment has equally likely outcomes. Let be “coin shows heads” and be “die shows a 5.” Then , , and . Indeed , so and are independent. That matches common sense: the coin doesn’t care what the die does.

Do not confuse independence with mutual exclusivity. If and are mutually exclusive () and both have non‑zero probability, then but , so they are not independent. In fact, knowing one occurred completely rules out the other—strong dependence.

Independence can be extended to more than two events, but the definition needs care.

📝 Section Recap: Two events are independent if the chance of both occurring is the product of their individual chances: . This captures the idea that learning one event gives no information about the other.

Pairwise and Mutual Independence#

For three events , we might first think they are independent if every pair is independent. That condition—pairwise independence—is weaker than full independence.

Mutual independence of events means that for every subset of the events, the probability of the intersection equals the product of the individual probabilities. For three events, this requires:

and the triple condition:

Pairwise independence only guarantees the first three equations; it does not force the triple product to factor. Mutual independence demands that all subset intersections factor, not just pairs.

A classic counterexample: two coins. Toss two fair coins. Define

: first coin heads,
: second coin heads,
: both coins show the same face (both heads or both tails).

Then . The pairs:
;
;
.
So the events are pairwise independent. But , while . The triple fails, so the events are not mutually independent. Intuitively, if you already know and happened, you know happened for sure—so together they give information.

When we say a set of events is “independent” without qualification, we always mean mutually independent. This distinction becomes especially important in reliability analysis and probabilistic proofs.

📝 Section Recap: Mutual independence requires that the probability of the intersection of any subset of events equals the product of their probabilities. Pairwise independence is not enough; triple and higher‑order intersections must also factor.

Independence of Random Variables#

A random variable is a number that depends on the outcome of an experiment—for example, the result of a die roll, or a person’s height. To talk about independence of two random variables, we look at events that can be described only in terms of (such as “ equals 5” or “”) and events that can be described only in terms of .

Independent random variables: and are independent if for any (reasonable) sets and ,

In plain words: every event about is independent of every event about .

For discrete random variables this condition is equivalent to the joint probability mass function (pmf) factoring:

for all possible values and .

For continuous random variables independence means that the joint probability density function (pdf) splits into the product of the marginal densities:

for all (except possibly on a set of measure zero—a set so small it does not affect any probability).

In either case the idea is the same: knowing the value of gives you no information about the outcome of , and vice versa. When we model real phenomena, we often assume variables are independent to simplify analysis—for example, successive coin flips or the error terms in a linear regression.

Independence can be extended to more than two random variables: a collection is mutually independent if for any sets ,

In terms of densities, the joint pdf (or pmf) splits completely into the product of the marginals.

📝 Section Recap: Two random variables are independent when the probability of any joint event factors as the product of the individual event probabilities. For discrete variables, the joint pmf is the product of marginal pmfs; for continuous variables, the joint pdf is the product of marginal pdfs.

Product Measures for Independent Experiments#

Many experiments consist of two or more unrelated sub‑experiments, like rolling a die and flipping a coin. Probability theory has a clean way to describe such independent combinations: the product probability measure.

Suppose we have two independent experiments with sample spaces and and probability measures and . The natural sample space for the combined experiment is the set of all ordered pairs . We want a probability measure on these pairs that reflects independence: the outcome of the first experiment has no bearing on the second. For any set and , we define

This rule extends to all events built from such rectangles and defines the product measure .

Discrete product measure. If both experiments are discrete, with probability mass functions and , the joint pmf for the product experiment is simply

Any event’s probability is the sum of these products over the event. For instance, two independent fair die rolls have for each ordered pair. The probability that the sum is is the sum over pairs where , which equals .

Continuous product measure. For independent random variables and with marginal densities and , the joint density is the product:

Then the probability that falls in a region is obtained by integrating the joint density over :

This factorization is both a definition of independence and a practical tool. It lets us build complex models by multiplying simpler ones.

The product‑measure construction generalises to any finite or countable number of independent experiments. In each case, probabilities (or densities) multiply, which is a direct consequence of mutual independence.

📝 Section Recap: When experiments are independent, the probability measure on the combined sample space is the product measure, obtained by multiplying the probabilities of the individual outcomes. In the discrete case, the joint pmf is the product of marginals; in the continuous case, the joint pdf is the product of marginals.

Summary#

We began with a simple but profound question: how do we update probability when we learn that something happened? That gave us conditional probability. We then saw how to break a probability into conditional pieces using a partition—the Law of Total Probability. Bayes’ formula turned the tables, letting us compute the “reverse” conditional probability and update our prior beliefs. Independence then arrived as the extreme case where conditioning changes nothing. We distinguished pairwise from mutual independence and extended the idea to random variables, culminating in the product‑measure construction for independent experiments. These tools are the workhorses of probability: they let you reason about partial information, combine evidence, and build models out of independent pieces.

Key idea	What it means (plain English)	Why it matters
Conditional probability	The chance of happening if we already know happened. It equals .	It lets you update probabilities when you receive new information—essential for any kind of learning from data.
Law of Total Probability	The overall probability of an event is a weighted average of its conditional probabilities across a partition, weighted by the partition’s probabilities.	Breaks complex problems into simpler conditional chunks; the basis for many inference and simulation methods.
Bayes’ formula	Reverses a conditional probability: .	The mathematical engine for updating beliefs. Powers medical diagnosis, spam filters, machine learning classifiers, and scientific reasoning.
Independence (events)	Two events are independent if —knowing one gives no information about the other.	Allows huge simplifications in models; often a reasonable assumption in real‑world systems.
Mutual vs. pairwise independence	Mutual independence requires that all subset intersections factor; pairwise only requires each pair.	Many probability results rely on full mutual independence. Pairwise alone can give deceptively wrong conclusions in multi‑variable settings.
Independence of random variables	and are independent if events about and are independent. In practice, their joint pmf or pdf factors: .	Justifies multiplying probabilities for separate measurements; the foundation for the behaviour of sums, averages, and limit theorems.
Product probability measure	For independent experiments, the probability of a combined outcome is the product of the individual probabilities.	Formalises how to combine independent sources of randomness; gives us the joint distribution from marginals.