Chapter 1: Foundations of Continuous-Time Markov Processes#

Imagine a frog hopping between lily pads in a pond. It rests on one pad for a while, then suddenly jumps to another. The moment it jumps next and which pad it picks both depend only on the pad it is sitting on now — not on where it has been before. This is the essence of a continuous-time Markov process — a random walk through time where the future depends only on the present state. In this chapter we build the core ideas needed to describe, analyse, and compare such processes.

The Big Picture#

Why do we need continuous-time models? Many real-world systems — like a computer server flipping between busy and idle, chemical reactions changing concentration, or animal populations moving — don’t change at fixed clock ticks. They evolve at random moments. A Continuous-Time Markov Chain (CTMC) captures this well: the process can switch state at any time, and the rules are described by a matrix of rates, not probabilities.

In this chapter we will define the process, learn how to write its transition probabilities, find the intensity matrix , derive the equations that describe how probabilities change, and see the simple picture of exponential holding times and an embedded discrete‑time chain that drives the jumps. By the end, you’ll have the basic tools to work with CTMCs.

What Is a Continuous-Time Markov Process?#

A continuous-time Markov process is a random process that changes over time, taking values in a set (the state space). The key property: the future depends only on the current state, not on history. More precisely, for any times and any states ,

We usually assume the chain is time‑homogeneous, meaning the right‑hand side depends only on the time difference , not on the starting time . Then we can define the transition probability from state to state in time as

For each fixed , the form a matrix . Immediately we have (the identity matrix), because after no time the process has not moved.

A crucial consequence of the Markov property is the Chapman–Kolmogorov equation:

In words: to go from state at time to state at time , the process must pass through some intermediate state at time ; the probability of that chain of events is . This equation ties the whole probabilistic structure together.

Time‑homogeneous CTMC: A Markov process where the transition probabilities depend only on the time difference , not on the absolute starting time.

We usually think of the states as discrete, labelled or a finite set like . The transition probabilities are smooth functions of time (usually differentiable), which will let us work with rates and derivatives.

📝 Section Recap: A continuous-time Markov process is defined by transition probabilities that depend only on the time difference, not on the past. The Chapman–Kolmogorov equation captures the memoryless property: to go from to in time , you must pass through some state at time .

The Intensity Matrix: How Fast Transitions Happen#

Transition probabilities are useful, but they don’t show how fast changes happen. To see the speed, we zoom in on a very tiny time interval . For a small , the chance of jumping from state to a different state is roughly proportional to :

where is the transition rate from to . More precisely,

For the diagonal entries, we need the rate at which the process stays in the same state over a tiny interval. Since , we define

The matrix

is called the intensity matrix (or -matrix, or generator). Its off‑diagonal entries are non‑negative, the diagonal entries are negative or zero, and every row sums to :

The number is the total rate of leaving state . It tells you how quickly, on average, the process exits that state.

Intensity matrix : A matrix storing all the transition rates. For a very short time , the chance of jumping from state to state (with ) is about . The rows sum to zero.

Let’s look at a simple but important example: a two‑state chain where the process jumps from state to state at rate and from state to state at rate . Then

For a very short time , the chance of moving from to is about , and the chance of staying in is about . This matches the idea of a “rate”.

A useful fact (for a finite number of states) is that the transition probability matrix can be written as a matrix exponential:

This formula wraps all the behaviour over any time into the intensity matrix. The matrix exponential also makes sure that always satisfies the Chapman–Kolmogorov equation and that each row sums to (so it is a valid probability matrix).

📝 Section Recap: The intensity matrix gives the instantaneous rates of jumping between states. Its off‑diagonal entries are non‑negative, each row sums to zero, and is the total rate out of state . The whole transition matrix can be recovered as .

Kolmogorov’s Forward and Backward Equations#

Knowing that is nice, but we often want to work directly with differential equations that show how the transition probabilities change over time. These are the Kolmogorov equations, and they come in two flavours: forward and backward.

Start from the Chapman–Kolmogorov relation for a small . Subtract and divide by :

Now take the limit . Because (by definition of ), we obtain the Kolmogorov forward equation:

If we instead use , a similar limit gives the Kolmogorov backward equation:

Both equations hold under mild conditions; for a finite number of states they are always true. The forward equation tracks how probabilities spread out from the initial state by focusing on the final small time step. The backward equation fixes the final state and looks at the first tiny step from the start.

For the two‑state example with , the forward equations for the probability of being in state when starting in are

together with . Solving gives

The probability decays exponentially towards the long‑run values. The rate controls how fast the chain “forgets” its starting state.

Kolmogorov equations: Differential equations connecting the derivative of the transition matrix to the intensity matrix. Forward: ; backward: .

📝 Section Recap: Kolmogorov’s forward and backward equations describe how the transition probabilities change over time. They follow from the Chapman–Kolmogorov relation and the definition of the intensity matrix, and they let us compute by solving simple differential equations.

How Jumps Work: Exponential Holding Times and the Embedded Chain#

So far we have described CTMCs through their transition probabilities and the intensity matrix. But there is a more concrete picture: every time the process enters a state , it waits for a random holding time, and then it jumps to a different state. The waiting time and the next state follow simple rules.

First, the holding time in state , call it , has an exponential distribution with rate . That is,

The exponential distribution is memoryless: no matter how long you have already waited, the remaining waiting time still has the same exponential distribution. This memoryless property is exactly what makes the process Markov — the future doesn’t depend on how long the process has been lingering.

If , then the state is absorbing; the process never leaves it.

Second, when the holding time finally ends, the process jumps to a different state with probability

These probabilities define a discrete‑time Markov chain, called the embedded Markov chain, which records the order of states visited at the jump instants. The embedded chain never stays in the same state (its diagonal entries are ), and its transition matrix has entries given by these ratios.

So we have a clean decomposition:

CTMC = embedded discrete‑time Markov chain + exponential holding times.

You can build a CTMC from scratch this way: start with a discrete‑time chain (with zeros on the diagonal) and a set of positive holding rates for each state. Then set for and . This completely defines a continuous‑time Markov chain.

For example, consider a three‑state chain with states . Suppose from state the transition rates are and . Then . The holding time in state is exponential with rate (mean ). When the jump happens, the process moves to state with probability and to state with probability . That tells you everything about the behaviour from state .

The embedded chain tells you where you go next; the holding rates tell you how fast. This separation makes CTMCs much easier to understand and compare.

📝 Section Recap: A CTMC can be understood as a sequence of states visited at jump times (the embedded chain) together with exponentially distributed holding times with rates . The memoryless property of the exponential keeps the Markov property, and the next state is chosen independently according to the ratios .

Summary#

We’ve built the foundations of continuous‑time Markov chains. You now know that a CTMC is a memoryless process whose future depends only on the current state. The intensity matrix captures all the transition rates, and from we can write differential equations (the Kolmogorov equations) to find the transition probabilities at any time. Finally, the process has a simple physical picture: in each state it waits for an exponential holding time, then jumps according to an embedded discrete‑time chain. These ideas give you the basic toolkit for working with CTMCs.