Markov Processes for Stochastic Modeling
- 2nd Edition - June 3, 2013
- Author: Oliver Ibe
- Language: English
- Paperback ISBN:9 7 8 - 0 - 3 2 3 - 2 8 2 9 5 - 6
- Hardback ISBN:9 7 8 - 0 - 1 2 - 4 0 7 7 9 5 - 9
- eBook ISBN:9 7 8 - 0 - 1 2 - 4 0 7 8 3 9 - 0
Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of ma… Read more
Purchase options
Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of many systems including communications systems, transportation networks, image segmentation and analysis, biological systems and DNA sequence analysis, random atomic motion and diffusion in physics, social mobility, population studies, epidemiology, animal and insect migration, queueing systems, resource management, dams, financial engineering, actuarial science, and decision systems.
Covering a wide range of areas of application of Markov processes, this second edition is revised to highlight the most important aspects as well as the most recent trends and applications of Markov processes. The author spent over 16 years in the industry before returning to academia, and he has applied many of the principles covered in this book in multiple research projects. Therefore, this is an applications-oriented book that also includes enough theory to provide a solid ground in the subject for the reader.
- Presents both the theory and applications of the different aspects of Markov processes
- Includes numerous solved examples as well as detailed diagrams that make it easier to understand the principle being presented
- Discusses different applications of hidden Markov models, such as DNA sequence analysis and speech analysis.
Graduate and upper-level undergraduate students, researchers and practitioners working in Markov Processes.
Chapter 1: Basic Concepts
Review of Probability
Random Variables
Transform Methods
Bivariate Random Variables
Many Random Variables
Fubini’s Theorem
Sums of Independent Random Variables
Some Probability Distributions
Introduction to Stochastic Processes
Classification of Stochastic Processes
Characterizing a Stochastic Process
Stationary Stochastic Processes
Ergodic Stochastic Processes
Some Models of Stochastic Processes
Chapter 2: Introduction to Markov Processes
Introduction
Structure of Markov Processes
Strong Markov Property
Applications of Discrete-time Markov Processes
Applications of Continuous-time Markov Processes
Applications of Continuous-state Markov Processes
Chapter 3: Discrete-Time Markov Chains
Introduction
State Transition Probability Matrix
State Transition Diagrams
Classification of States
Limiting-State Probabilities
Sojourn Time
Transient Analysis of Discrete-Time Markov Chains
First Passage and Recurrence Times
Occupancy Times
Absorbing Markov Chains and the Fundamental Matrix
Reversible Markov Chains
Chapter 4: Continuous-Time Markov Chains
Introduction
Transient Analysis
Birth and Death Processes
First Passage Time
The Uniformization Method
Reversible Continuous-Time Markov Chains
Chapter 5: Markovian Queueing Systems
Introduction
Description of a Queueing System
The Kendall Notation
The Little’s Formula
The PASTA Property
The M/M/1 Queueing System
Examples of Other M/M Queueing Systems
M/G/1 Queue
G/M/1 Queue
Chapter 6: Markov Renewal Processes
Renewal Processes
The Renewal Equation
The Elementary Renewal Theorem
Random Incidence and Residual Time
Markov Renewal Process
Semi-Markov Processes
Markov Jump Processes
Chapter 7: Markovian Arrival Processes
Introduction
Overview of Matrix-Analytic Methods
Markovian Arrival Process
Batch Markovian Arrival Process
Markov-Modulated Poisson Process
Markov-Modulated Bernoulli Process
Sample Applications of MAP and Its Derivatives
Chapter 8: Random Walk
Introduction
The Two-Dimensional Random Walk
Random Walk as a Markov Chain
Symmetric Random Walk as a Martingale
Random Walk with Barriers
Gambler’s Ruin
First Return Times
First Passage Times
Maximum of a Random Walk
Correlated Random Walk
Continuous-time Random Walk
Sample Applications of Random Walk
Chapter 9: Brownian Motion and Diffusion Processes
Introduction
Brownian Motion
Introduction to Stochastic Calculus
Geometric Brownian Motion
Fractional Brownian Motion
Application of Brownian Motion to Option Pricing
Random Walk Approximation of Brownian Motion
The Ornstein-Uhlenbeck Process
Diffusion Processes
Examples of Diffusion Processes
Relationship Between the Diffusion Process and Random Walk
Chapter 10: Controlled Markov Processes
Introduction
Markov Decision Processes
Semi-Markov Decision Processes
Partially Observable Markov Decision Processes
Chapter 11: Hidden Markov Models
Introduction
HMM Basics
HMM Assumptions
Three Fundamental Problems
Solution Methods
Types of Hidden Markov Models
Hidden Markov Models with Silent States
Extensions of Hidden Markov Models
Other Extensions of HMM
Chapter 12: Markov Point Processes
Point Processes
Temporal Point Processes
Spatial Point Processes
Spatial-Temporal Point Processes
Operations on Point Processes
Marked Point Processes
Markov Point Processes
Markov Marked Point Processes
Applications of Markov Point Processes
- Edition: 2
- Published: June 3, 2013
- Language: English
OI
Oliver Ibe
Dr Ibe has been teaching at U Mass since 2003. He also has more than 20 years of experience in the corporate world, most recently as Chief Technology Officer at Sineria Networks and Director of Network Architecture for Spike Broadband Corp.
Affiliations and expertise
University of Massachusetts, Lowell, USARead Markov Processes for Stochastic Modeling on ScienceDirect