MARKOV CHAINS:LONG-RUN PROPERTIES OF MARKOV CHAINS

By -

LONG-RUN PROPERTIES OF MARKOV CHAINS

Steady-State Probabilities

While calculating the n-step transition probabilities for both the weather and inventory examples in Sec. 29.3, we noted an interesting feature of these matrices. If n is large enough (n = 5 for the weather example and n = 8 for the inventory example), all the rows of the matrix have identical entries, so the probability that the system is in each state j no longer depends on the initial state of the system. In other words, there is a limiting prob- ability that the system will be in each state j after a large number of transitions, and this probability is independent of the initial state. These properties of the long-run behavior of finite-state Markov chains do, in fact, hold under relatively general conditions, as sum- marized below.

INTRODUCTION TO OPERATIONS RESEARCH-0721INTRODUCTION TO OPERATIONS RESEARCH-0722INTRODUCTION TO OPERATIONS RESEARCH-0723

Thus, the probability of finding the process in a transient state after a large number of transitions tends to zero.

Expected Average Cost per Unit Time

The preceding subsection dealt with irreducible finite-state Markov chains whose states were ergodic (recurrent and aperiodic). If the requirement that the states be aperiodic is relaxed, then the limit

INTRODUCTION TO OPERATIONS RESEARCH-0724

where the TTj satisfy the steady-state equations given in the preceding subsection.

This result is important in computing the long-run average cost per unit time associated with a Markov chain. Suppose that a cost (or other penalty function) C(Xt) is incurred when the process is in state Xt at time t, for t = 0, 1, 2, . . . . Note that C(Xt) is a random variable that takes on any one of the values C(0), C(1), . . . , C(M) and that the function C(·) is independent of t. The expected average cost incurred over the first n periods is given by

INTRODUCTION TO OPERATIONS RESEARCH-0725

Application to the Inventory Example. To illustrate, consider the inventory example introduced in Sec. 29.1, where the solution for the TTj was obtained in an earlier subsection. Suppose the camera store finds that a storage charge is being allocated for each camera remaining on the shelf at the end of the week. The cost is charged as follows:

INTRODUCTION TO OPERATIONS RESEARCH-0726

Expected Average Cost per Unit Time for Complex Cost Functions

In the preceding subsection, the cost function was based solely on the state that the process is in at time t. In many important problems encountered in practice, the cost may also depend upon some other random variable.

For example, in the inventory example introduced in Sec. 29.1, suppose that the costs to be considered are the ordering cost and the penalty cost for unsatisfied demand (stor- age costs are so small they will be ignored). It is reasonable to assume that the number of cameras ordered to arrive at the beginning of week t depends only upon the state of the process Xt-1 (the number of cameras in stock) when the order is placed at the end of week t - 1. However, the cost of unsatisfied demand in week t will also depend upon the demand Dt. Therefore, the total cost (ordering cost plus cost of unsatisfied demand) for week t is a function of Xt-1 and Dt, that is, C(Xt-1, Dt).

Under the assumptions of this example, it can be shown that the (long-run) expected

INTRODUCTION TO OPERATIONS RESEARCH-0727

INTRODUCTION TO OPERATIONS RESEARCH-0728

The results of this subsection were presented only in terms of the inventory example. However, the (nonnumerical) results still hold for other problems as long as the following conditions are satisfied:

INTRODUCTION TO OPERATIONS RESEARCH-0729

Comments

Post a Comment

Popular posts from this blog

DUALITY THEORY:THE ESSENCE OF DUALITY THEORY

By -
Image
THE ESSENCE OF DUALITY THEORY Given our standard form for the primal problem at the left (perhaps after conversion from another form), its dual problem has the form shown to the right. Thus, with the primal problem in maximizatio n form, the dual problem is in minimization form instead. Furthermore, the dual problem uses exactly the same pa r amete r s as the primal problem, but in different locations, as summarized below. 1. The coefficients in the objective function of the primal problem are the right-hand sides of the functional constraints in the dual problem. 2. The right-hand sides of the functional constraints in the primal problem are the coef- ficients in the objective function of the dual problem. 3. The coefficients of a variable in the functional constraints of the primal problem are the coefficients in a functional constraint of the dual problem. To highlight the comparison, now look at these same two problems in matrix notation (as introduced at the beginn...
Read more

NETWORK OPTIMIZATION MODELS:THE MINIMUM SPANNING TREE PROBLEM

By -
Image
THE MINIMUM SPANNING TREE PROBLEM The minimum spanning tree problem bears some similarities to the main version of the shortest-path problem presented in the preceding section. In both cases, an undi r ecte d and connected network is being considered, where the given information includes some mea- sure of the positive length (distance, cost, time, etc.) associated with each link. Both problems also involve choosing a set of links that have the shortest total length among all sets of links that satisfy a certain property. For the shortest-path problem, this property is that the chosen links must provide a path between the origin and the destination. For the minimum spanning tree problem, the required property is that the chosen links must provide a path between ea c h pair of nodes. The minimum spanning tree problem can be summarized as follows: 1. You are given the nodes of a network but not the links. Instead, you are given the po- tential links and the positive length for each ...
Read more

INTEGER PROGRAMMING:THE BRANCH-AND-CUT APPROACH TO SOLVING BIP PROBLEMS

By -
TH E BRANCH-AND-CUT APPROACH TO SOLVING BIP PROBLEMS Integer programming has been an especially exciting area of OR since the mid-1980s be- cause of the dramatic progress being made in its solution methodology. Background To place this progress into perspective, consider the historical background. One big break- through had come in the 1960s and early 1970s with the development and refinement of the branch-and-bound approach. But then the state of the art seemed to hit a plateau. Rel- atively small problems (well under 100 variables) could be solved very efficiently, but even a modest increase in problem size might cause an explosion in computation time be- yond feasible limits. Little progress was being made in overcoming this exponential growth in computation time as the problem size was increased. Many important problems arising in practice could not be solved. Then came the next breakthrough in the mid-1980s, with the introduction of the b r an c h- and-cu t approach to sol...
Read more