Introduction To The Numerical Solution Of Markov Chains

A cornerstone of applied probability, Markov chains can be used to help model how plants grow, chemicals react, and atoms diffuse--and applications are increasingly being found in such areas as engineering, computer science, economics, and education. To apply the techniques to real problems, however, it is necessary to understand how Markov chains can be solved numerically. In this book, the first to offer a systematic and detailed treatment of the numerical solution of Markov chains, William Stewart provides scientists on many levels with the power to put this theory to use in the actual world, where it has applications in areas as diverse as engineering, economics, and education. His efforts make for essential reading in a rapidly growing field. Here Stewart explores all aspects of numerically computing solutions of Markov chains, especially when the state is huge. He provides extensive background to both discrete-time and continuous-time Markov chains and examines many different numerical computing methods--direct, single-and multi-vector iterative, and projection methods. More specifically, he considers recursive methods often used when the structure of the Markov chain is upper Hessenberg, iterative aggregation/disaggregation methods that are particularly appropriate when it is NCD (nearly completely decomposable), and reduced schemes for cases in which the chain is periodic. There are chapters on methods for computing transient solutions, on stochastic automata networks, and, finally, on currently available software. Throughout Stewart draws on numerous examples and comparisons among the methods he so thoroughly explains.

Intersecting two large research areas - numerical analysis and applied probability/queuing theory - this book is a self-contained introduction to the numerical solution of structured Markov chains, which have a wide applicability in queuing theory and stochastic modeling and include M/G/1 and GI/M/1-type Markov chain, quasi-birth-death processes, non-skip free queues and tree-like stochastic processes. Written for applied probabilists and numerical analysts, but accessible toengineers and scientists working on telecommunications and evaluation of computer systems performances, it provides a systematic treatment of the theory and algorithms for important families of structured Markov chains and a thorough overview of the current literature.The book, consisting of nine Chapters, is presented in three parts. Part 1 covers a basic description of the fundamental concepts related to Markov chains, a systematic treatment of the structure matrix tools, including finite Toeplitz matrices, displacement operators, FFT, and the infinite block Toeplitz matrices, their relationship with matrix power series and the fundamental problems of solving matrix equations and computing canonical factorizations. Part 2 deals with the description andanalysis of structure Markov chains and includes M/G/1, quasi-birth-death processes, non-skip-free queues and tree-like processes. Part 3 covers solution algorithms where new convergence and applicability results are proved. Each chapter ends with bibliographic notes for further reading, and the bookends with an appendix collecting the main general concepts and results used in the book, a list of the main annotations and algorithms used in the book, and an extensive index.

Papers presented at a workshop held January 1990 (location unspecified) cover just about all aspects of solving Markov models numerically. There are papers on matrix generation techniques and generalized stochastic Petri nets; the computation of stationary distributions, including aggregation/disaggregation.

Besides the investigation of general chains the book contains chapters which are concerned with eigenvalue techniques, conductance, stopping times, the strong Markov property, couplings, strong uniform times, Markov chains on arbitrary finite groups (including a crash-course in harmonic analysis), random generation and counting, Markov random fields, Gibbs fields, the Metropolis sampler, and simulated annealing. With 170 exercises.

This book provides an undergraduate-level introduction to discrete and continuous-time Markov chains and their applications, with a particular focus on the first step analysis technique and its applications to average hitting times and ruin probabilities. It also discusses classical topics such as recurrence and transience, stationary and limiting distributions, as well as branching processes. It first examines in detail two important examples (gambling processes and random walks) before presenting the general theory itself in the subsequent chapters. It also provides an introduction to discrete-time martingales and their relation to ruin probabilities and mean exit times, together with a chapter on spatial Poisson processes. The concepts presented are illustrated by examples, 138 exercises and 9 problems with their solutions.

The field of applied probability has changed profoundly in the past twenty years. The development of computational methods has greatly contributed to a better understanding of the theory. A First Course in Stochastic Models provides a self-contained introduction to the theory and applications of stochastic models. Emphasis is placed on establishing the theoretical foundations of the subject, thereby providing a framework in which the applications can be understood. Without this solid basis in theory no applications can be solved. Provides an introduction to the use of stochastic models through an integrated presentation of theory, algorithms and applications. Incorporates recent developments in computational probability. Includes a wide range of examples that illustrate the models and make the methods of solution clear. Features an abundance of motivating exercises that help the student learn how to apply the theory. Accessible to anyone with a basic knowledge of probability. A First Course in Stochastic Models is suitable for senior undergraduate and graduate students from computer science, engineering, statistics, operations resear ch, and any other discipline where stochastic modelling takes place. It stands out amongst other textbooks on the subject because of its integrated presentation of theory, algorithms and applications.

This book gives a systematic treatment of singularly perturbed systems that naturally arise in control and optimization, queueing networks, manufacturing systems, and financial engineering. It presents results on asymptotic expansions of solutions of Komogorov forward and backward equations, properties of functional occupation measures, exponential upper bounds, and functional limit results for Markov chains with weak and strong interactions. To bridge the gap between theory and applications, a large portion of the book is devoted to applications in controlled dynamic systems, production planning, and numerical methods for controlled Markovian systems with large-scale and complex structures in the real-world problems. This second edition has been updated throughout and includes two new chapters on asymptotic expansions of solutions for backward equations and hybrid LQG problems. The chapters on analytic and probabilistic properties of two-time-scale Markov chains have been almost completely rewritten and the notation has been streamlined and simplified. This book is written for applied mathematicians, engineers, operations researchers, and applied scientists. Selected material from the book can also be used for a one semester advanced graduate-level course in applied probability and stochastic processes.

This book considers a class of ergodic finite controllable Markov's chains. The main idea behind the method, described in this book, is to develop the original discrete optimization problems (or game models) in the space of randomized formulations, where the variables stand in for the distributions (mixed strategies or preferences) of the original discrete (pure) strategies in the use. The following suppositions are made: a finite state space, a limited action space, continuity of the probabilities and rewards associated with the actions, and a necessity for accessibility. These hypotheses lead to the existence of an optimal policy. The best course of action is always stationary. It is either simple (i.e., nonrandomized stationary) or composed of two nonrandomized policies, which is equivalent to randomly selecting one of two simple policies throughout each epoch by tossing a biased coin. As a bonus, the optimization procedure just has to repeatedly solve the time-average dynamic programming equation, making it theoretically feasible to choose the optimum course of action under the global restriction. In the ergodic cases the state distributions, generated by the corresponding transition equations, exponentially quickly converge to their stationary (final) values. This makes it possible to employ all widely used optimization methods (such as Gradient-like procedures, Extra-proximal method, Lagrange's multipliers, Tikhonov's regularization), including the related numerical techniques. In the book we tackle different problems and theoretical Markov models like controllable and ergodic Markov chains, multi-objective Pareto front solutions, partially observable Markov chains, continuous-time Markov chains, Nash equilibrium and Stackelberg equilibrium, Lyapunov-like function in Markov chains, Best-reply strategy, Bayesian incentive-compatible mechanisms, Bayesian Partially Observable Markov Games, bargaining solutions for Nash and Kalai-Smorodinsky formulations, multi-traffic signal-control synchronization problem, Rubinstein's non-cooperative bargaining solutions, the transfer pricing problem as bargaining.