Markov Processes for Stochastic Modeling

Chapter 1: Basic Concepts

Review of Probability

Random Variables

Transform Methods

Bivariate Random Variables

Many Random Variables

Fubini’s Theorem

Sums of Independent Random Variables

Some Probability Distributions

Introduction to Stochastic Processes

Classification of Stochastic Processes

Characterizing a Stochastic Process

Stationary Stochastic Processes

Ergodic Stochastic Processes

Some Models of Stochastic Processes

Chapter 2: Introduction to Markov Processes

Introduction

Structure of Markov Processes

Strong Markov Property

Applications of Discrete-time Markov Processes

Applications of Continuous-time Markov Processes

Applications of Continuous-state Markov Processes

Chapter 3: Discrete-Time Markov Chains

Introduction

State Transition Probability Matrix

State Transition Diagrams

Classification of States

Limiting-State Probabilities

Sojourn Time

Transient Analysis of Discrete-Time Markov Chains

First Passage and Recurrence Times

Occupancy Times

Absorbing Markov Chains and the Fundamental Matrix

Reversible Markov Chains

Chapter 4: Continuous-Time Markov Chains

Introduction

Transient Analysis

Birth and Death Processes

First Passage Time

The Uniformization Method

Reversible Continuous-Time Markov Chains

Chapter 5: Markovian Queueing Systems

Introduction

Description of a Queueing System

The Kendall Notation

The Little’s Formula

The PASTA Property

The M/M/1 Queueing System

Examples of Other M/M Queueing Systems

M/G/1 Queue

G/M/1 Queue

Chapter 6: Markov Renewal Processes

Renewal Processes

The Renewal Equation

The Elementary Renewal Theorem

Random Incidence and Residual Time

Markov Renewal Process

Semi-Markov Processes

Markov Jump Processes

Chapter 7: Markovian Arrival Processes

Introduction

Overview of Matrix-Analytic Methods

Markovian Arrival Process

Batch Markovian Arrival Process

Markov-Modulated Poisson Process

Markov-Modulated Bernoulli Process

Sample Applications of MAP and Its Derivatives

Chapter 8: Random Walk

Introduction

The Two-Dimensional Random Walk

Random Walk as a Markov Chain

Symmetric Random Walk as a Martingale

Random Walk with Barriers

Gambler’s Ruin

First Return Times

First Passage Times

Maximum of a Random Walk

Correlated Random Walk

Continuous-time Random Walk

Sample Applications of Random Walk

Chapter 9: Brownian Motion and Diffusion Processes

Introduction

Brownian Motion

Introduction to Stochastic Calculus

Geometric Brownian Motion

Fractional Brownian Motion

Application of Brownian Motion to Option Pricing

Random Walk Approximation of Brownian Motion

The Ornstein-Uhlenbeck Process

Diffusion Processes

Examples of Diffusion Processes

Relationship Between the Diffusion Process and Random Walk

Chapter 10: Controlled Markov Processes

Introduction

Markov Decision Processes

Semi-Markov Decision Processes

Partially Observable Markov Decision Processes

Chapter 11: Hidden Markov Models

Introduction

HMM Basics

HMM Assumptions

Three Fundamental Problems

Solution Methods

Types of Hidden Markov Models

Hidden Markov Models with Silent States

Extensions of Hidden Markov Models

Other Extensions of HMM

Chapter 12: Markov Point Processes

Point Processes

Temporal Point Processes

Spatial Point Processes

Spatial-Temporal Point Processes

Operations on Point Processes

Marked Point Processes

Markov Point Processes

Markov Marked Point Processes

Applications of Markov Point Processes