Fundamentals of Stationary Stochastic Processes

Have you ever wondered how we can predict tomorrow’s temperature, or the next value of a stock index, when the world seems so unpredictable? The secret lies in finding patterns in the noise—patterns that stay the same over time. This chapter lays the foundation for all dynamic modeling: we learn to describe the memory of a time series by first making sure that memory doesn’t keep changing its own rules.

The Big Picture#

Time series analysis deals with sequences of observations taken over time—like daily rainfall, quarterly sales, or brain signals measured every second. Before we can build a model to forecast or explain such sequences, we need a way to talk about how the past influences the present. But if the rules of influence keep shifting, any model we build will quickly become useless. The idea of stationarity ensures that the basic statistical properties of the series—its average level, its variability, and how values relate across time—stay the same. Once we have that stability, we can meaningfully measure dependence with tools like the autocorrelation function, and then construct simple, powerful models like autoregressive and moving average processes. By the end of this chapter, you will understand how to measure how strongly past values affect future ones and how to express that mathematically, setting the stage for everything that follows.

What Makes a Time Series Stationary?#

Imagine you record the temperature in a room every hour, and the room has a steady thermostat. The temperature values will bounce around a constant average, the size of the ups and downs will be roughly the same, and the link between today’s and yesterday’s temperature won’t change from one month to the next. That’s the heart of stationarity: the statistical pattern of the series does not change when you slide the time axis. Formally, there are two levels.

Strict stationarity is the strongest form: the whole joint distribution of any set of values must be exactly the same no matter where you look in time. That’s hard to check in practice. Instead, we almost always use a simpler requirement called weak stationarity (or second-order stationarity), which only cares about the first two moments:

Weak stationarity: A time series is weakly stationary if its mean is constant for all , its variance is constant for all , and the covariance depends only on the lag , not on time .

So, a weakly stationary series has:

• A mean that does not drift over time,
• A variance that does not explode or shrink,
• A dependence structure that is the same whether you look at the beginning or the end of the series.

Think of it like a calm ocean: the wave heights (variance) are steady, the sea level (mean) remains fixed, and the way one wave influences the next doesn’t change as you sail from east to west.

Why do we care so much about weak stationarity? Because most of the models we build—AR, MA, ARMA—assume it. If our data are not stationary, we can often transform them (by differencing, removing a trend, or taking logs) to make them so. Stationarity gives us a common language: when we talk about the “autocorrelation at lag 2,” that number has a clear meaning that stays the same throughout the series.

📝 Section Recap: A stationary time series has constant mean, constant variance, and a dependence structure that only depends on the time separation, not on absolute time. This stability is the prerequisite for meaningful modeling and forecasting.

White Noise: The Simplest Building Block#

The most basic stationary process is white noise. The name comes from an analogy with white light, which contains all visible colors mixed together equally. A white noise process has no memory—each observation is uncorrelated with all other observations, and they all share the same mean and variance.

Formally, a sequence is a white noise process if:

• for all ,
• for all ,
• for any .

Often we also assume the are independent and identically distributed (i.i.d.), and sometimes Gaussian. In that case we call it Gaussian white noise. But the core idea is zero correlation across time.

White noise is the seed from which all more interesting time series can be grown. Models like AR and MA take a white noise input and shape it into a process with memory. If you can see that a series is essentially white noise, you know there is nothing to forecast: the best prediction for tomorrow is zero (the mean), no matter what happened today. This makes white noise an important benchmark for checking whether a model has captured all the dependence in the data.

White noise: A sequence of uncorrelated random variables with constant mean (often zero) and constant variance.

📝 Section Recap: White noise is a completely memoryless, stationary process that serves as the fundamental building block and the benchmark for “no structure left.” All serial dependence in real data is measured against this baseline.

Measuring Memory: The Autocorrelation Function (ACF)#

Once a series is stationary, how do we measure how much today’s value tells us about tomorrow’s? The most natural tool is correlation: the same idea you know from scatterplots, but now applied to a series with itself at different time lags. This is called the autocorrelation function (ACF).

For a weakly stationary process with mean , the autocovariance at lag is defined as

Because of stationarity, does not depend on —it only depends on the gap . When , is just the variance .

The autocorrelation at lag is the autocovariance scaled by the variance:

This gives a dimensionless number between and , exactly like an ordinary correlation coefficient. The sequence of autocorrelations for different lags is called the autocorrelation function (ACF).

Autocorrelation : The correlation between an observation and the observation time steps earlier, for a stationary series. It measures the linear predictability across that lag.

How do you read an ACF? For white noise, for all . For a series with positive persistence, like daily temperature, might be large and positive (today’s temperature is highly correlated with yesterday’s), and the ACF decays slowly as the lag increases. For a series that oscillates, might alternate in sign.

It’s useful to visualize the ACF as a bar chart (often called a correlogram). In practice, we estimate from the data using the sample autocorrelation, often denoted . The pattern of these estimated correlations gives crucial hints about what kind of model might fit well. For example, a sharp cutoff in the ACF after a certain lag points toward a particular class of models.

📝 Section Recap: The ACF measures linear dependence as a function of lag. It translates “Does yesterday help predict today?” into a single correlation coefficient, providing our first diagnostic of a time series’ memory.

Peeking Inside Dependence: The Partial Autocorrelation Function (PACF)#

The ACF tells us the total correlation between and . But some of that correlation may be indirect—passed along through the intermediate lags. For example, if is correlated with both and , then the ACF at lag 2 will include not only the direct link between and but also the chain . To isolate the direct relationship, we use the partial autocorrelation function (PACF).

Think of it this way: suppose we want to know whether the day before yesterday () still helps predict today () after we already know yesterday (). That extra correlation, with the influence of removed, is the partial autocorrelation at lag 2. Formally, is the correlation between and after removing the linear effects of the intervening values .

Partial autocorrelation : The correlation between and after removing the linear effects of all intervening observations. It measures the direct predictive contribution of the -th lag.

For a stationary process, the PACF can be estimated from the data, and its pattern is a powerful model-identification tool. In particular, for a pure autoregressive (AR) process of order , the theoretical PACF cuts off to zero after lag . That is, for , and for . This means that if you see a sample PACF that becomes essentially zero after, say, lag 2, a pure AR(2) model is a strong candidate.

By contrast, the ACF of an AR process tails off gradually rather than cutting off. The interplay between the ACF and PACF shapes is a classic method for guessing the appropriate model order before we start estimation.

📝 Section Recap: The PACF filters out indirect associations and reveals the unique contribution of each past lag. A sudden drop to zero in the PACF after a certain lag is the hallmark of an autoregressive process of that order.

The Autoregressive Model AR(p)#

Now we can build our first real model. An autoregressive process of order , written AR(), says: the current value is a weighted sum of the previous values plus a fresh white noise shock. In other words, the series uses its own past to predict its present.

The mathematical formulation is:

where is white noise with zero mean and variance , and is a constant. Often we subtract the mean to simplify: define , then the centered form is

The coefficients determine how the past echoes into the present. For the process to be stationary, these coefficients must satisfy stability conditions. For an AR(1) this simply means ; otherwise the series would drift or blow up. For higher orders, the requirement is that all roots of the characteristic polynomial lie outside the unit circle. Roughly speaking, the feedback from the past must die out, not explode. For an AR(1) with , each shock dies out slowly; with , the series oscillates quickly around zero.

Why is the PACF so informative here? In an AR() process, once you have included the first lags, any additional lag has no direct linear influence—its partial autocorrelation is zero. That’s exactly why the theoretical PACF cuts off after lag . The ACF, on the other hand, decays smoothly because the remote past still exerts an indirect effect through the chain of ’s.

Consider an AR(2) with and zero mean. The equation reads . If you simulate it, you will see a wandering, slightly smoothed pattern. The ACF shows a gradual decline, but the PACF shows large spikes at lags 1 and 2, then drops to near zero. That visual match is how we recognize an AR(2).

📝 Section Recap: An AR(p) model expresses the current observation as a linear function of its own most recent values plus white noise. The PACF is the go-to tool for detecting the order , because it cuts off precisely at that lag for a pure AR process.

Moving Average Processes#

Autoregressive models are intuitive: the past directly enters the equation. But there is another fundamental building block: the moving average (MA) process. Despite the name, this has nothing to do with averaging a window. Instead, the current value is a weighted sum of the current and past white noise shocks.

A moving average process of order , denoted MA(), is given by

where is white noise with variance . The constant is the mean of the series. Each coefficient tells us how much a shock periods ago still affects today’s value. Because the ’s are uncorrelated, the process is always stationary (as long as the ’s are finite).

What does the ACF of an MA() look like? Since depends only on the last shocks, the correlation between and will be nonzero only for . That is, the theoretical ACF cuts off after lag . The PACF, by contrast, tails off gradually (the opposite of the AR case). This mirror-image relationship is a key identification clue: a sharp drop in the ACF suggests an MA model, while a sharp drop in the PACF suggests an AR.

For a concrete MA(1): . If , a positive shock today continues to affect tomorrow’s value positively, then disappears. The ACF will show a spike at lag 1 and zero everywhere else. The PACF will decay geometrically. This behavior is easy to spot in sample correlograms.

Moving average processes are especially natural for modeling series where the observed values are the result of accumulated unobserved impulses—think of economic indicators that react to a series of small, independent news items.

📝 Section Recap: An MA(q) process writes the observation as a finite combination of the most recent white noise shocks. Its ACF cuts off after lag , making it easy to identify, while the PACF tails off.

The ARMA Model: Putting It Together#

Real-world time series are rarely pure AR or pure MA. Often the data contain both short-term shock propagation (MA-like behavior) and longer-term autoregressive feedback (AR-like behavior). The autoregressive moving average (ARMA) model combines the two into a single, flexible framework.

The ARMA() model is defined by

where is white noise. Subtracting the mean yields the centered form. This equation says: today’s value is a mix of the past observed values and the current and past unobserved shocks.

ARMA(): A stationary time series model that includes both autoregressive and moving average components. The orders and respectively count how many past observations and how many past shocks enter the equation.

Stationarity of an ARMA process depends only on the AR part. The moving average part is always finite, so the condition is the same as before: the AR characteristic polynomial must have all roots outside the unit circle. An ARMA process is invertible if the MA polynomial is similarly well-behaved, which guarantees that the current observation can be expressed as a convergent infinite autoregression. We won’t worry about that now.

The ACF and PACF for a mixed ARMA model are more complex: neither cuts off cleanly if both and are positive. Typically both functions tail off, so choosing the orders and from correlograms alone can be tricky. That’s why later chapters introduce estimation and information criteria. For now, understanding the pure AR and MA cases gives you the pattern language you need.

Think of an ARMA model as describing a damped pendulum that receives random kicks. The autoregressive part captures the swinging dynamics (its tendency to return toward center with some momentum), while the moving average part captures how successive random kicks are smoothed. Almost any stationary time series with a reasonably short memory can be approximated by an ARMA process of sufficiently high order—a deep result (Wold’s theorem) states that any purely non‑deterministic stationary process can be written as an infinite MA, and often a finite‑order ARMA is an excellent approximation.

By combining AR and MA structures, we gain simplicity: a low‑order ARMA(1,1) may fit data that would require a high‑order pure AR or pure MA, meaning fewer parameters and more reliable estimates.

📝 Section Recap: The ARMA model marries autoregressive and moving average components into a compact, powerful description of stationary time series. It captures both direct feedback from past observations and the lingering influence of past shocks, forming the core dynamic model upon which much of modern time series analysis is built.

Summary#

You have just taken the first important step into dynamic modeling. We started with stationarity, the idea that a time series must have a stable statistical personality before we can trust any patterns we find. White noise gave us the simplest possible stationary process—a clean slate with no memory. We then built two measurement tools—the ACF and the PACF—that show us how strongly the past signals the present, and whether that link is direct or just passed through intermediate steps. Those tools led us to two fundamental model families: autoregressive (AR) processes, where the past values themselves predict the future, and moving average (MA) processes, where the accumulation of recent shocks shapes the current observation. Finally we combined them into the ARMA model, a flexible, compact description of stationary series that will serve as your workhorse.

Every concept here is designed to be intuitive: think of the ACF as asking “How much does yesterday matter?” and the PACF as “Does the day before yesterday still matter, even after I know yesterday?” The clean cutoff patterns you now recognize—PACF chopping off for AR, ACF chopping off for MA—are like diagnostic fingerprints. Keep these patterns in mind, because they will guide you whenever you meet a new time series dataset.