Operations Research Models and Methods / Computation / Unit Title

Markov Chain



Creating a Model

Economic Analysis

Transient Analysis

Steady State Analysis

Probability Analysis

First Passage Probabilities


Absorbing Probabilities

Return to Stochastic Analysis

The Stochastic Analysis Add-in performs a wide range of computations associated with Markov Chains.


Selecting the Markov Chain item under Stochastic Analysis, provides the opportunity to construct a Markov Chain Model. We describe below the various analyses possible for the example whose state-transition diagram is shown above. The analysis types are listed at the left, and the description of an analysis can be reached directly by clicking on the item.

Example: Light Bulb Replacement

A bowling alley has a sign made entirely of light bulbs. There are 1000 bulbs in the sign. The manager is concerned about the maintenance of the sign. He wonders how many of the bulbs must be replaced in a monthly service call and how much should he budget for maintenance of the sign.

Historical data and probability analysis determine the probabilities of replacement based on the age of a bulb. This information is shown by the matrix below.

Each row of the matrix describes what might occur for a bulb of a particular age. For example, the row labeled New indicates that if a new bulb is inserted in a socket during one of the monthly maintenance inspections, the probability that it will fail during the month and be replaced with a new bulb at the next inspection is 0.5. The probability that it will not fail and survive to an age of one month is also 0.5. This poor quality seems to be rather extreme, but we exaggerate for illustration. The remaining rows indicate similar data for other ages. In every case the bulb is replaced with a new one or ages by one month. For the example, we assume the bulb is always replaced after it is four months old.

The names New, 1-mo, 2-mo., etc. are the states of the system. At a monthly inspection the bulb must be in one of the states. The matrix is called the transition matrix because it shows the probability of transition from each state to every other state. We call this a Markov process when the transition probabilities depend only on the state of the system. The figure appearing at the beginning of this article is called the State Transition Diagram and represents the same information as the transition matrix.

Creating a Model

Updated 3/27/01
Operations Research Models and Methods

by Paul A. Jensen and Jon Bard, University of Texas, Copyright by the Authors