Markov Models in Continuous Time

When an insurance contract covers health, disability, or death, a person’s status can change at any moment — not just at the end of a year. To model this continuous uncertainty, we use continuous-time Markov models. This chapter shows how to turn instantaneous rates into probabilities we can use. We start with the simple idea that the future depends only on the present, and build up to the Kolmogorov equations.

The Big Picture#

Imagine a policyholder who can be healthy, disabled, or dead. Over time they may jump between these states, and the timing of those jumps is unpredictable. We want to answer questions like: if someone is healthy now, what is the chance they will be disabled in five years, or dead next week? Continuous-time Markov models give us the tools: we describe how the system leaves each state through intensities (instantaneous rates), and then solve differential equations to find the full probability distribution over time. This chapter builds that framework step by step, from the memoryless Markov property to the matrix exponential that ties it all together.

The Markov Property in Continuous Time#

The foundation of everything that follows is the Markov property: given the present state, the past tells us nothing extra about the future.

Formally, for a stochastic process taking values in a set of states (for example, 1=Healthy, 2=Disabled, 3=Dead), the process is Markov if for any times and any states , we have

All that matters is where you are right now, not how you got there.

Markov process: A random process whose future behaviour depends only on the current state, not on the path taken to reach it.

A real‑life analogy is a game of snakes and ladders. If you land on square 42, the next dice roll determines your future — the fact that you were on square 7 ten minutes ago is irrelevant. Similarly, a healthy policyholder’s future health transitions depend only on the fact they are healthy today, not on whether they were ill last year.

In this chapter we work mostly with time-homogeneous processes. This means the transition rules do not change with time: the probability of moving from state to state over an interval of length depends only on , not on the starting age. This simplification makes the mathematics cleaner and is a good first approximation for many insurance products.

📝 Section Recap: A continuous-time Markov process has no memory — the next move depends solely on the current state. When the rules are constant over time we call the process time-homogeneous.

Transition Probabilities: From State to State#

For any model with several states, the most natural quantity is the probability of being in a particular state after some time has passed. For a time-homogeneous Markov process we write

for any starting time . This is the probability that a person who is in state now will be in state after a further time .

These probabilities satisfy two simple but important conditions:

1. Non-negativity: for every and every .
2. Sum over all destinations: . From any starting state you must end up in one of the possible states (assuming the set includes all possible destinations, including an absorbing state like Death).

Transition probability : The chance that a process currently in state will be in state after time .

For a tiny time interval , we can approximate the behaviour from the instantaneous chances of jumping out of the current state. This leads us to the next building block: transition intensities.

📝 Section Recap: Transition probabilities describe the full probabilistic evolution over a fixed time horizon. They are non‑negative and, for each starting state, sum to 1 over all possible future states.

Transition Intensities: The Instantaneous Heartbeat of the Model#

Transition probabilities give the long‑term view. Transition intensities (also called forces of transition) describe the immediate risk of moving from one state to another. They are like the one‑step transition probabilities in a discrete‑time chain, but in continuous time.

For a time-homogeneous process, the intensity of moving from state to state () is the constant defined by

In words: if you are in state , then in the next tiny instant the probability of a jump to is approximately .

Transition intensity : The instantaneous rate at which a process leaves state directly for state . It is a rate per unit time, not a probability.

From state you might be able to jump to several other states. The total rate of leaving , often called the total force of decrement, is

If we wait for a short time , the chance of leaving for any destination is roughly , and the chance of staying in is approximately .

Example – a simple disability model.
States: 1 = Healthy, 2 = Disabled, 3 = Dead.
Transition intensities:

• : healthy → disabled rate.
• : healthy → dead rate (death while healthy).
• : disabled → healthy recovery rate.
• : disabled → dead rate.

Note there is no jump back from Dead; state 3 is absorbing. For a healthy person the total exit rate is .

Because intensities are instantaneous rates, they can be larger than 1 (unlike probabilities). For example, if per year, then over a short month ( year) the probability of becoming disabled is about , which is indeed a probability.

📝 Section Recap: Transition intensities are the instantaneous hazard rates for each possible move. They capture the moment‑by‑moment risk and are the basic parameters of the continuous-time Markov model.

The Chapman–Kolmogorov Equations: Linking Steps Together#

Now that we have both probabilities over finite time and instantaneous intensities, we need a bridge between them. The Chapman–Kolmogorov equations do exactly that: they connect probabilities over different time intervals.

For any time-homogeneous Markov process, the probability of going from to in time can be broken down by considering all possible intermediate states at time :

This is simply the law of total probability: to end up in after , you must pass through some state at time .

Chapman–Kolmogorov equations: Equations that link transition probabilities over different time intervals by summing over all possible intermediate states.

You can think of it as “splicing” the future. If you want the probability of being disabled 10 years from now, you can first compute where the person might be in 3 years (healthy, disabled, or dead) and then, from each of those possibilities, compute the chance of ending up disabled in the remaining 7 years. Summing those conditional chances gives the overall probability.

Matrix form. Let be the matrix with entries . Then the Chapman–Kolmogorov equations become the elegant semigroup property:

This matrix multiplication automatically performs the sum over intermediate states. For instance, the entry of the product is indeed .

This semigroup property is the key that allows us to differentiate and obtain the Kolmogorov differential equations in the next section.

📝 Section Recap: The Chapman–Kolmogorov equations say that probabilities over long periods can be built up by chaining together probabilities over shorter periods. In matrix form, , a property that underpins the link to intensities.

Kolmogorov’s Differential Equations: The Forward and Backward View#

With the Chapman–Kolmogorov equations in hand, we can find how the transition probabilities change as we nudge time forward or backward. The result is a pair of differential equations that connect the probabilities with the transition intensities.

Define the intensity matrix (also called the generator matrix) whose off‑diagonal entries are the and whose diagonal entries are (so each row sums to zero):

Now consider a short time . From the definition of intensities, for very small , where is the identity matrix. Plug this into the Chapman–Kolmogorov equation and expand to first order in :

Rearranging and letting yields the Kolmogorov forward differential equation:

Alternatively, if we write and expand first, we obtain the Kolmogorov backward differential equation:

Both are valid, and their solutions agree for time-homogeneous processes. The forward equation looks at how later changes depend on the current state distribution (“what will happen to me?”), while the backward equation looks at how the starting state influences the future (“if I start here, where can I go?”).

In component form, the forward equation for a specific target state is

The first sum adds the probability of arriving from other states into , while the subtracted term removes the probability of leaving (because if you are already in and leave, you are no longer there). This is a neat balance equation.

Kolmogorov forward equation: A differential equation that describes how the probability of being in a given state evolves forward in time, with flows in from other states and flows out to other states.

Kolmogorov backward equation: A differential equation that describes how the probability depends on the initial state, showing how small changes in the starting state propagate forward.

These equations are the centrepiece of continuous-time Markov modelling. With them, if we know the intensity matrix , we can — at least in principle — compute all transition probabilities by solving the differential system with the initial condition .

📝 Section Recap: The Kolmogorov forward and backward equations translate the instantaneous intensities into differential equations for the transition probabilities. The forward equation tracks flows into and out of each state; the backward equation looks at how the starting state affects the probabilities.

Occupancy Probabilities and Exponential Solutions#

For many insurance calculations, what really matters is the probability of occupying a set of states — for example, the chance of being alive (healthy or disabled), regardless of which non‑dead state. These occupancy probabilities are simply sums of the appropriate :

When there are only two states — say, Alive and Dead — the model collapses to a single survival probability. If the death intensity is constant , the forward equation for the probability of being alive at time , starting alive, is

which has the familiar exponential solution

This is why survival probabilities often look like exponential decay in simple models. The constant intensity is exactly the force of mortality from classical life contingencies.

For a general multiple state model with a constant intensity matrix , the system of forward equations with has the unique solution

Matrix exponential : The solution to the differential equation ; it packages all transition probabilities into a single matrix function.

While computing a matrix exponential by hand is rarely done — we use numerical methods — the formula tells us that the whole probabilistic future is set by the intensity matrix . The exponential form also guarantees that satisfies the Chapman–Kolmogorov semigroup property: .

Example: disability model with constant intensities.
Take , , , (per year). The intensity matrix is

The probability of staying healthy for one year, , is not just because recovery from disability can bring you back to healthy. The matrix exponential captures these indirect paths. So even with constant intensities, occupancy probabilities can have richer shapes than a simple exponential when there are return paths.

Occupancy probabilities are the direct input for premium and reserve calculations — they answer questions like “What fraction of a cohort starting healthy will be alive (in any state) in 20 years?”

📝 Section Recap: Occupancy probabilities sum the probabilities of being in a desired set of states. With constant intensities, the full matrix of transition probabilities is given by the matrix exponential , which neatly satisfies both Kolmogorov equations and the Chapman‑Kolmogorov conditions.

Summary#

We have built the core ideas of continuous-time Markov models from the ground up. Starting from the memoryless Markov property, we defined transition probabilities and their instantaneous rates, the intensities. The Chapman–Kolmogorov equations let us splice time intervals, and from there we derived the Kolmogorov forward and backward differential equations — the key link between intensities and probabilities. Finally, with constant intensities the whole system solves to a matrix exponential, giving us a complete description of occupancy probabilities over any time horizon. This framework is the foundation for pricing and reserving in health, disability, and life insurance.