Chapter 2: Model Building with ARIMA Processes#

Many real-world time series—stock prices, monthly sales, global temperatures—wander and drift. They do not hover around a fixed mean like the stationary processes we met earlier. In this chapter you will learn a classic toolkit that handles these wandering series by turning them into stationary ones, then builds a simple model to capture the leftover patterns. The result is a flexible family called ARIMA, and by the end you will be able to take a raw dataset, find a suitable ARIMA model, check that it works well, and produce forecasts.

The Big Picture#

When a series has a trend that pulls it away from any constant level, the usual autocorrelation estimates break down—they can suggest false relationships. The ARIMA approach solves this by first turning a non‑stationary series into a stationary one via differencing, then modelling the stationary remainder with an ARMA structure. Once we have a model, we reverse the differencing to obtain forecasts of the original data. This chapter walks you through the full Box‑Jenkins cycle: identifying plausible model orders, estimating parameters, checking whether the model adequately captures the structure, and finally forecasting.

From Stationarity to Non‑Stationarity: The Idea of Differencing#

Imagine measuring the height of a sunflower every morning. The sequence grows steadily—the measurements are not clustered around a fixed average, so the series is non‑stationary. If we compute the day‑to‑day changes instead, those increments will fluctuate around a roughly constant value (the daily growth rate). That new series of changes is much more likely to be stationary.

In time‑series language we call this operation differencing. For a series the first difference is simply

If the original series contains a linear trend, will typically wander around a constant mean. Some series need a second round: , which can remove a quadratic trend. We denote the order of differencing by , so a series that becomes stationary after differences is called integrated of order , written .

Differencing: The process of subtracting the previous observation from the current one to remove trend‑related non‑stationarity. A series that requires such subtractions to become stationary is integrated of order .

Why go through this trouble? Because all our familiar tools—autocorrelation, partial autocorrelation, ARMA models—rely on stationarity. Differencing is the bridge that brings a drifting series back into the world where those tools work.

📝 Section Recap: Differencing transforms a non‑stationary series into a stationary one by looking at changes between observations, allowing us to apply stationary modelling techniques to data that originally had trends.

The Anatomy of an ARIMA Model#

An ARIMA model has three ingredients:

• AR (AutoRegressive) part: the current value depends on a few of its own past values.
• I (Integrated) part: the series has been differenced times to make it stationary.
• MA (Moving Average) part: the current value depends on a few past random shocks.

We write the full model as ARIMA(p, d, q), where , , and are non‑negative integers specifying the order of each component.

Start with the familiar ARMA(p, q) for a stationary series :

Here is white noise—a sequence of uncorrelated random shocks with zero mean and constant variance . The ’s are the autoregressive coefficients, and the ’s are the moving‑average coefficients. This ARMA model captures the stationary dynamics.

Now suppose our original series is not stationary but becomes stationary after differences. Let

The ARIMA(p, d, q) model is simply

ARIMA(p,d,q): The ‑times differenced series follows an ARMA(p,q) process. Equivalently, is an integrated ARMA process.

For example, ARIMA(1,1,0) means we difference once, and the differenced series follows an AR(1):

A random walk is the special case ARIMA(0,1,0) with no AR or MA terms: , which is just , pure white noise after differencing.

The notation packs a lot of information: the three numbers tell you exactly how many differences to take and how many past values and past errors to include.

📝 Section Recap: An ARIMA(p,d,q) model applies an ARMA(p,q) structure to the ‑times differenced series, giving us a unified way to describe both the trend‑removal step and the stationary dynamics.

Phase 1: Identifying the Model Order#

Before we can fit an ARIMA model we need a good guess for , , and . This identification stage uses plots and summary statistics, not formal estimation.

Choosing . Look at a time plot of the original series. If it shows a clear trend, try . If the first‑differenced series still shows a trend, consider . Rarely do we need for real data. A more rigorous check: a stationary series has an autocorrelation function (ACF) that decays reasonably quickly. If the ACF of the raw data dies out very slowly, differencing is indicated. You can also apply a unit‑root test like the augmented Dickey‑Fuller test, but visual inspection combined with the ACF is often enough to pick .

Choosing and . Once we have a candidate stationary series , we examine its ACF and partial autocorrelation function (PACF).

• The ACF at lag measures the correlation between and . For a pure MA() process, the ACF cuts off to zero after lag .
• The PACF at lag measures the correlation between and after removing the influence of the intervening lags. For a pure AR() process, the PACF cuts off after lag .

In practice, theory gives us clean patterns:

We look at the sample ACF and PACF plots and try to match the pattern. This is more art than science, and we often select a few candidate combinations to test in the next stage.

📝 Section Recap: Identification uses differencing to choose and the ACF/PACF of the differenced series to suggest plausible orders and , guided by the cut‑off or tail‑off behaviour.

Phase 2: Estimating the Parameters#

Given candidate orders , we now need actual numbers for the coefficients and , plus the white‑noise variance . The go‑to method is maximum likelihood estimation (MLE). Under the assumption that the white‑noise innovations are independent and normally distributed, MLE finds the parameter values that make the observed data most probable.

Because the likelihood for ARIMA models is a complicated function of the parameters, we rely on numerical optimisation algorithms behind the scenes. Software packages (such as the statsmodels library in Python or the forecast package in R) do the heavy lifting. For simple models, one can also use conditional sum of squares—a fast least‑squares approximation that ignores early observations—but MLE is the standard choice.

You rarely do this by hand. However, the key output to examine is the estimated coefficients, their standard errors, and whether they are statistically different from zero. Large standard errors or very high correlation between parameter estimates may indicate a model with too many parameters or a mistake in choosing , , or .

📝 Section Recap: Given model orders, we estimate the AR and MA coefficients (and the noise variance) by maximum likelihood, leaving us with a fully specified, fitted ARIMA model.

Phase 3: Diagnostic Checking — Are the Residuals White Noise?#

A good model should capture all the systematic pattern in the data, leaving behind only unpredictable noise. Diagnostic checking examines the residuals

where are the fitted values from the model. If the model is correct, the residuals should behave like white noise—uncorrelated, with zero mean and constant variance.

We check three things:

1. Residual time plot: Should look like random scatter around zero with no obvious trends or changing variance.
2. ACF of residuals: For white noise, all autocorrelations should be close to zero. We use the Ljung‑Box test (or the similar Box‑Pierce test) to formally test whether the first residual autocorrelations are jointly zero. A large p‑value (say ) supports the null hypothesis of white noise.
3. Distributional checks: A histogram or Q‑Q plot can reveal gross non‑normality, though mild departures are often tolerable.

If the residuals show significant correlation, we go back to identification—maybe a higher order or is needed, or we missed a seasonal pattern. This cycle of “identify → estimate → diagnose” is the heart of the Box‑Jenkins methodology.

White Noise: A sequence of random variables that are uncorrelated, have zero mean, and constant variance. It is the “leftover” that a good model cannot predict.

📝 Section Recap: Diagnostic checking verifies that the fitted model has extracted all predictable structure; the residuals should be uncorrelated and constant in variance, which we assess via time plots, ACF plots, and the Ljung‑Box test.

Forecasting with ARIMA Models#

Once we trust our model, we can produce forecasts. The idea is simple: use the estimated ARIMA equation to project future values, and then undo the differencing to return to the original scale.

Suppose we have an ARIMA(1,1,0) model:

To forecast , we first forecast the difference:

Then, because , we get

For multi‑step forecasts we iterate the ARMA equation on the differenced scale and cumulate the predicted differences. The forecast function for a differenced model will typically not revert to a fixed mean; it can drift linearly or even quadratically, depending on .

Forecast intervals come from the estimated standard error of the predictions, which grows as we look further ahead—reflecting our increasing uncertainty. The intervals are usually constructed under the assumption of normally distributed innovations, though they often work reasonably even with modest departures.

📝 Section Recap: ARIMA forecasts are obtained by iterating the fitted ARMA model on the differenced scale and then accumulating the predicted differences back to the original data scale, with interval widths that widen with the forecast horizon.

A Special Case: The Local Trend Model and ARIMA(0,1,1)#

A beautiful connection exists between a simple structural model and an ARIMA specification. Consider a local trend model, also known as a random walk plus noise:

and and are independent. This model says the observed series is a slowly wandering level plus an erratic measurement error. The level changes by a random step each period.

If we take the first difference of , we get

Notice that the right‑hand side is a combination of two independent white‑noise sequences, which produces an MA(1) process. In fact, one can show that follows an MA(1) with a negative moving‑average coefficient (provided the variances are positive). Therefore, the local trend model is exactly an ARIMA(0,1,1).

This equivalence explains why the ARIMA(0,1,1) model, often called the exponential smoothing model, is so popular for series that seem to move without any obvious AR or cyclic pattern. The model captures a slowly adjusting level, and its forecasts are simply an exponentially weighted moving average of past observations—a technique forecasters have used for decades.

📝 Section Recap: The local trend model, with a random walk level plus observation noise, is mathematically equivalent to an ARIMA(0,1,1), linking structural thinking with the Box‑Jenkins framework.

Summary#

You now have a complete, practical pipeline for building models when a time series drifts. Differencing transforms the drift into stationarity, the ARMA machinery captures the remaining short‑term dependencies, and a careful diagnostic check keeps the model honest. Once the model passes inspection, you can generate forecasts with honest uncertainty bands. The ARIMA family may look like a simple string of numbers—(p,d,q)—but it is a powerful method that has been used successfully in economics, engineering, and environmental science for decades.