Limited Offer

# Markov Processes for Stochastic Modeling

- 2nd Edition - May 22, 2013
- Author: Oliver Ibe
- Language: English
- Hardback ISBN:9 7 8 - 0 - 1 2 - 4 0 7 7 9 5 - 9
- Paperback ISBN:9 7 8 - 0 - 3 2 3 - 2 8 2 9 5 - 6
- eBook ISBN:9 7 8 - 0 - 1 2 - 4 0 7 8 3 9 - 0

Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of ma… Read more

## Purchase options

## Institutional subscription on ScienceDirect

Request a sales quoteMarkov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of many systems including communications systems, transportation networks, image segmentation and analysis, biological systems and DNA sequence analysis, random atomic motion and diffusion in physics, social mobility, population studies, epidemiology, animal and insect migration, queueing systems, resource management, dams, financial engineering, actuarial science, and decision systems.

Covering a wide range of areas of application of Markov processes, this second edition is revised to highlight the most important aspects as well as the most recent trends and applications of Markov processes. The author spent over 16 years in the industry before returning to academia, and he has applied many of the principles covered in this book in multiple research projects. Therefore, this is an applications-oriented book that also includes enough theory to provide a solid ground in the subject for the reader.

- Presents both the theory and applications of the different aspects of Markov processes
- Includes numerous solved examples as well as detailed diagrams that make it easier to understand the principle being presented
- Discusses different applications of hidden Markov models, such as DNA sequence analysis and speech analysis.

Chapter 1: Basic Concepts

Review of Probability

Random Variables

Transform Methods

Bivariate Random Variables

Many Random Variables

Fubini’s Theorem

Sums of Independent Random Variables

Some Probability Distributions

Introduction to Stochastic Processes

Classification of Stochastic Processes

Characterizing a Stochastic Process

Stationary Stochastic Processes

Ergodic Stochastic Processes

Some Models of Stochastic Processes

Chapter 2: Introduction to Markov Processes

Introduction

Structure of Markov Processes

Strong Markov Property

Applications of Discrete-time Markov Processes

Applications of Continuous-time Markov Processes

Applications of Continuous-state Markov Processes

Chapter 3: Discrete-Time Markov Chains

Introduction

State Transition Probability Matrix

State Transition Diagrams

Classification of States

Limiting-State Probabilities

Sojourn Time

Transient Analysis of Discrete-Time Markov Chains

First Passage and Recurrence Times

Occupancy Times

Absorbing Markov Chains and the Fundamental Matrix

Reversible Markov Chains

Chapter 4: Continuous-Time Markov Chains

Introduction

Transient Analysis

Birth and Death Processes

First Passage Time

The Uniformization Method

Reversible Continuous-Time Markov Chains

Chapter 5: Markovian Queueing Systems

Introduction

Description of a Queueing System

The Kendall Notation

The Little’s Formula

The PASTA Property

The M/M/1 Queueing System

Examples of Other M/M Queueing Systems

M/G/1 Queue

G/M/1 Queue

Chapter 6: Markov Renewal Processes

Renewal Processes

The Renewal Equation

The Elementary Renewal Theorem

Random Incidence and Residual Time

Markov Renewal Process

Semi-Markov Processes

Markov Jump Processes

Chapter 7: Markovian Arrival Processes

Introduction

Overview of Matrix-Analytic Methods

Markovian Arrival Process

Batch Markovian Arrival Process

Markov-Modulated Poisson Process

Markov-Modulated Bernoulli Process

Sample Applications of MAP and Its Derivatives

Chapter 8: Random Walk

Introduction

The Two-Dimensional Random Walk

Random Walk as a Markov Chain

Symmetric Random Walk as a Martingale

Random Walk with Barriers

Gambler’s Ruin

First Return Times

First Passage Times

Maximum of a Random Walk

Correlated Random Walk

Continuous-time Random Walk

Sample Applications of Random Walk

Chapter 9: Brownian Motion and Diffusion Processes

Introduction

Brownian Motion

Introduction to Stochastic Calculus

Geometric Brownian Motion

Fractional Brownian Motion

Application of Brownian Motion to Option Pricing

Random Walk Approximation of Brownian Motion

The Ornstein-Uhlenbeck Process

Diffusion Processes

Examples of Diffusion Processes

Relationship Between the Diffusion Process and Random Walk

Chapter 10: Controlled Markov Processes

Introduction

Markov Decision Processes

Semi-Markov Decision Processes

Partially Observable Markov Decision Processes

Chapter 11: Hidden Markov Models

Introduction

HMM Basics

HMM Assumptions

Three Fundamental Problems

Solution Methods

Types of Hidden Markov Models

Hidden Markov Models with Silent States

Extensions of Hidden Markov Models

Other Extensions of HMM

Chapter 12: Markov Point Processes

Point Processes

Temporal Point Processes

Spatial Point Processes

Spatial-Temporal Point Processes

Operations on Point Processes

Marked Point Processes

Markov Point Processes

Markov Marked Point Processes

Applications of Markov Point Processes

- No. of pages: 514
- Language: English
- Edition: 2
- Published: May 22, 2013
- Imprint: Elsevier
- Hardback ISBN: 9780124077959
- Paperback ISBN: 9780323282956
- eBook ISBN: 9780124078390

OI

### Oliver Ibe

*Markov Processes for Stochastic Modeling*on ScienceDirect