Markov Chains

Models random transitions, probabilistic evolution, and stationary equilibrium in discrete systems.

Conceptual context

Many systems change randomly between discrete states: a particle hops between regions, a molecule changes conformation, or a population moves between classes. A Markov chain simplifies such evolution by assuming that the next state depends on the present state rather than the entire past. The key object is $P$, which redistributes probability and determines whether $p_i(n)$ forgets the initial condition and approaches $\pi_{i}^{*}$.

🟢 Essential level

A Markov chain describes random transitions among discrete states. At each step, $P$ gives the probabilities of remaining in the same state or moving elsewhere. Starting from $\pi_{n}$, one obtains $\pi_{n1}$, and repeated updates determine $p_i(n)$ over $n$. If probabilities stop changing, the chain reaches $\pi^{*}$, whose components are $\pi_{i}^{*}$. This is statistical equilibrium, not microscopic rest. Individual trajectories may continue jumping while the overall distribution remains unchanged. Convergence is not automatic. It depends on communication between states and on the absence of rigid periodic cycles that sustain oscillations.

The essential distinction is between an observed state and a probability distribution. One trajectory occupies one state, whereas the chain describes expected frequencies across many realizations or over long times. This separation is necessary for interpreting probability, equilibrium, and recurrence correctly.

Confusing the two levels is a common mistake: observing that one particular particle stays in the same state for several steps does not contradict the existence of a stationary distribution, because the distribution describes the average behavior of many trajectories, not the fate of a single one.

🔵 Formal level

Basic formulas

Distribution evolution is written as:

Distribution evolution

$\pi_{n}$ is multiplied by $P$ to produce $\pi_{n1}$. Components must remain nonnegative and normalized.

A stationary distribution satisfies:

Stationary distribution

$\pi^{*}$ remains invariant under one transition, and each component defines $\pi_{i}^{*}$. Existence does not automatically imply uniqueness or convergence from every initial distribution.

Real student questions

Does a stationary distribution mean the system stops changing? No. Trajectories change while ensemble proportions remain constant.

Do all chains converge? No. Reducibility and periodicity may prevent convergence.

What does an entry of $P_n$ mean? It is the accumulated probability of moving between two states after several steps.

Why is $\tau_{i}$ useful? It measures how frequently a state reappears in the long run.

Is one trajectory enough? Only when it is long enough and the chain is ergodic.

Final synthesis

Markov chains show how local transition rules create probabilistic evolution, equilibrium, and recurrence. Convergence depends on both the transition matrix and the communication structure among states.

Reliable interpretation also requires checking normalization, state definitions, and sampling length.