Probability Theory for Econometrics

Economic data is messy. Stock prices jump, wages vary, and survey responses take on only a few values. To make sense of this randomness, we need a clear language for describing uncertainty — that language is probability theory. In this chapter, you’ll build the toolkit that every econometrician uses: random variables, distributions, moments, and the key distributions that appear in hypothesis testing. Don’t worry — we’ll take it step by step, with plenty of intuition.

The Big Picture#

How can we describe and quantify the randomness in economic data? Probability theory gives us a set of concepts to model uncertainty. Understanding these concepts is key because every estimator, test statistic, and confidence interval in econometrics rests on them. By the end of this chapter, you’ll be able to talk about the average, spread, and shape of a variable’s distribution, how two variables move together, and the special bell‑shaped curve that shows up everywhere.

Random Variables and Probability Distributions#

A random variable is a numerical outcome of a random process. Think of it as a rule that gives a number to each possible result of an experiment. For example, the number of children in a randomly chosen household is a random variable; the daily return on a stock index is another one.

Random variable: A variable whose value is determined by a chance process.

Random variables come in two main kinds.

• Discrete random variables can take only a countable list of values. The number of children (0, 1, 2, …) is discrete.
• Continuous random variables can take any value in an interval. The exact height of a person or the time until the next bus are continuous.

To describe how likely each outcome is, we use a probability distribution. For a discrete random variable, the distribution is given by a probability mass function (PMF):

It tells us the probability that the random variable equals a particular value . The probabilities must sum to 1 across all possible values.

Probability mass function (PMF): A function that gives the probability of each possible value for a discrete random variable.

For a continuous random variable, the probability of any single point is zero — there are infinitely many possible values. Instead, we use a probability density function (PDF) . The probability that falls in an interval is the area under the curve:

The total area under the PDF equals 1. The height is not a probability by itself; it’s a density — you multiply by a tiny width to get an approximate probability.

Probability density function (PDF): A function used for continuous random variables; the area under the curve over an interval gives the probability that the variable lies in that interval.

Another useful idea is the cumulative distribution function (CDF):

It works for both discrete and continuous variables. For a continuous variable, is the area under the PDF up to ; for a discrete variable, it is the sum of the PMF up to .

Cumulative distribution function (CDF): The probability that a random variable is less than or equal to a given value.

Imagine a lottery machine that spits out a number between 0 and 100. The PDF could be a flat line (uniform distribution), meaning every small range of numbers is equally likely. The CDF would climb steadily from 0 to 1. These functions are the basic building blocks for describing any random economic quantity.

📝 Section Recap: Random variables assign numbers to random outcomes, and their probability distributions (PMF for discrete, PDF for continuous) tell us how likely each outcome is. The CDF gives the probability of being below a threshold.

Expected Value and Moments#

Once we have a distribution, we want to summarise its main features. The most important summary is the centre, which we call the expected value or mean. It is the long‑run average if we could repeat the random process many times.

For a discrete variable:

For a continuous variable:

Think of a fair six‑sided die. Each face has probability , so the expected value is

You never roll a 3.5, but if you rolled the die thousands of times, the average of your rolls would approach 3.5.

Expected value ( or ): The probability‑weighted average of all possible values of a random variable; the long‑run average.

The expected value tells us nothing about how spread out the values are. For that we use the variance:

It is the average squared distance from the mean. A larger variance means the variable is more spread out. The square root of the variance is the standard deviation , which is in the same units as and is easier to interpret.

Variance ( or ): The expected squared deviation from the mean; measures the spread of a distribution.

Standard deviation (): The square root of the variance; a typical distance from the mean.

For the die, , and the variance works out to about 2.92, so the standard deviation is about 1.71. Most rolls will be within 1.71 of 3.5.

We also care about the shape. Skewness captures asymmetry:

A symmetric distribution (like the bell curve) has skewness zero. A positive skew means the right tail is longer than the left — think of income distributions, where a few very high incomes pull the average above the median. A negative skew means the left tail is longer.

Skewness: A measure of the asymmetry of a distribution; positive skew indicates a longer right tail.

Kurtosis tells us how heavy the tails are — how likely extreme values are compared to a normal bell curve:

A normal distribution has kurtosis 3. Often we look at excess kurtosis (kurtosis minus 3). Financial returns frequently have excess kurtosis greater than 0, meaning “fat tails” — big moves happen more often than a normal distribution would predict.

Kurtosis: A measure of the tail weight of a distribution; higher kurtosis means more probability in the tails.

These are all examples of moments: the ‑th raw moment is , and the ‑th central moment is . The expected value is the first raw moment, the variance is the second central moment, skewness uses the third, and kurtosis uses the fourth.

📝 Section Recap: The expected value is the centre, the variance and standard deviation measure spread, skewness describes asymmetry, and kurtosis tells us about tail thickness. Together, these moments summarise the shape of any distribution.

Relationships Between Variables#

Economic variables rarely move alone. We need tools to describe how two random variables, say and , are connected.

The joint distribution gives the probability of any pair of outcomes. For discrete variables, the joint PMF is . For continuous variables, the joint PDF gives the density over the plane; the probability that falls in a region is the double integral of over that region.

Joint distribution: The probability distribution of two (or more) random variables considered together.

From the joint distribution we can get the marginal distribution of one variable by summing (or integrating) over the other:

The marginal distribution of ignores entirely — it is just the ordinary distribution of we discussed earlier.

Marginal distribution: The probability distribution of one variable obtained by averaging out the other variable(s) from the joint distribution.

Often we want to know the distribution of given that we already know takes a particular value . That is the conditional distribution:

For instance, the conditional distribution of wages given education level tells us how wages are spread among people with exactly 12 years of schooling.

Conditional distribution: The distribution of one variable when the value of another variable is held fixed.

Two variables are independent if knowing one gives no information about the other. Formally, and are independent if the joint distribution factors into the product of the marginals:

Equivalently, the conditional distribution of given does not depend on . Independence is a very strong condition, but it simplifies many calculations.

Independence: Two random variables are independent if the probability of any joint outcome is the product of their individual probabilities.

To measure the linear relationship between two variables, we use covariance:

If and tend to move together (above‑average goes with above‑average ), the covariance is positive. If they move oppositely, it is negative. The covariance depends on the units of and , so it is hard to compare across different pairs. That is why we standardise it to get the correlation:

Correlation always lies between and . A correlation of means a strong positive linear relationship; means no linear relationship (though there could be a non‑linear one).

Covariance: A measure of how two variables move together linearly; the expected product of their deviations from their means.

Correlation (): A standardised version of covariance that falls between –1 and 1; it measures the strength and direction of a linear relationship.

In practice, we never know the true population covariance or correlation. We estimate them from a sample of paired observations . The sample covariance is

We divide by (not ) to get an unbiased estimator of the population covariance. The sample correlation is

where and are the sample standard deviations. Just like , is between –1 and 1.

Sample covariance (): An estimate of the population covariance computed from data, using in the denominator.

Sample correlation (): An estimate of the population correlation; it measures the linear association in the sample data.

Imagine you collect data on years of education and hourly wage for 500 workers. You would compute to see how strongly they are related. A positive near 0.5 might indicate that more education tends to go with higher wages, but the relationship is far from perfect.

📝 Section Recap: Joint distributions describe two variables together; marginal and conditional distributions let us zoom in on one variable. Independence means no relationship. Covariance and correlation measure linear co‑movement; sample versions estimate these quantities from data.

The Normal Distribution and Its Relatives#

The normal distribution is the most famous distribution in statistics. Its bell‑shaped curve appears whenever a variable is the sum of many small, independent influences — a result known as the Central Limit Theorem. In econometrics, we often assume that error terms or certain estimators follow a normal distribution, at least approximately.

A normal random variable with mean and variance is written . Its PDF is

The curve is symmetric around , and about 68% of the probability lies within of the mean, 95% within .

Normal distribution: A continuous, symmetric, bell‑shaped distribution completely described by its mean and variance .

If we subtract the mean and divide by the standard deviation, we get a standard normal variable . Tables and software give probabilities for , and any normal probability can be converted to a standard normal problem.

Three other distributions are especially important because they describe the behaviour of test statistics in hypothesis testing. All are derived from the normal.

The chi‑squared distribution arises when we square and sum independent standard normal variables. If are independent , then

The parameter is the degrees of freedom. The distribution is skewed to the right, with mean and variance . As grows, it becomes more symmetric and bell‑shaped. Chi‑squared variables appear when we estimate variances or test multiple restrictions at once.

Chi‑squared distribution (): The distribution of a sum of squared independent standard normal variables; used for variance‑related tests.

The Student’s t‑distribution is a ratio that shows up when we standardise a sample mean with an estimated standard deviation. If and are independent, then

The ‑distribution looks like a standard normal but has fatter tails, especially when (the degrees of freedom) is small. As , it approaches . In practice, when we test a single coefficient (e.g., the effect of education on wages) and we estimate the error variance from the data, the test statistic follows a ‑distribution under the null hypothesis.

Student’s t‑distribution (): A bell‑shaped distribution with heavier tails than the normal; used for testing means when the variance is estimated.

The F‑distribution is the ratio of two independent chi‑squared variables, each divided by its degrees of freedom. If and are independent, then

The ‑distribution is right‑skewed and takes only positive values. It is used to test whether several coefficients are jointly zero (for example, testing if both education and experience matter for wages) or to compare the fit of two models.

F‑distribution (): The distribution of a ratio of two independent chi‑squared variables scaled by their degrees of freedom; used for testing multiple linear restrictions.

All these distributions are connected: a ‑statistic squared with degrees of freedom is an variable. In econometrics, you will meet them constantly as the sampling distributions of your estimators and test statistics.

📝 Section Recap: The normal distribution is the benchmark bell curve. The chi‑squared, , and distributions are derived from normals and form the backbone of hypothesis testing, each with a specific role: chi‑squared for variance, for single‑coefficient tests, and for joint tests.

Summary#

We have covered the language of uncertainty that econometrics relies on. Random variables and their distributions let us model anything from coin flips to stock returns. Moments — expected value, variance, skewness, and kurtosis — summarise the centre, spread, and shape of a distribution. When we look at two variables together, joint, marginal, and conditional distributions describe their connections, while covariance and correlation measure linear relationships. Finally, the normal distribution and its relatives (chi‑squared, , and ) give us the precise probability statements we need for testing economic hypotheses. These ideas are the foundation for everything that follows.