MDP in Finance

MDP Theory

Introduction

The Markov Decision Process is the sequence of random variables () which describes the stochastic evolution
of the system states. Of course the distribution of () depends on the chosen actions.

• denotes the state space of the system. A state is the information which is available for the controller at time . Given this information an action has to be selected.

• denotes the action space. Given a specific state at time , a certain subclass of actions may only be admissible.

• is a stochastic transition kernel which gives the probability that the next state at time is in the set B if the current state is and action a is taken at time .

• gives the (discounted) one-stage reward of the system at time if the current state is and action a is taken

• gives the (discounted) terminal reward of the system at the end of the planning horizon.

A control is a sequence of decision rules () with where determines for each possible state the next action at time . Such a sequence is called policy or strategy. Formally the Markov Decision Problem is given by

• Types of MDP problems:

  • finite horizon () vs. infinite horizon ()
  • complete state observation vs. partial state observation
  • problems with constraints vs. without constraints
  • total (discounted) cost criterion vs. average cost criterion

• Research questions:

  • Does an optimal policy exist?
  • Has it a particular form?
  • Can an optimal policy be computed efficiently?
  • Is it possible to derive properties of the optimal value function analytically?

Applications: Consumption Problem

• Consumption Problem
  Suppose there is an investor with given initial capital. At the beginning of each of periods she can decide how much of the capital she consumes and how much she invests into a risky asset. The amount she consumes is evaluated by a utility function as well as the terminal wealth. The remaining capital is invested into a risky asset where we assume that the investor is small and thus not able to influence the asset price and the asset is liquid. How should she consume/invest in order to maximize the sum of her expected discounted utility?

• Cash Balance or Inventory Problem
  Imagine a company which tries to find the optimal level of cash over a finite number of periods. We assume that there is a random stochastic change in the cash reserve each period (due to withdrawal or earnings). Since the firm does not earn interest from the cash position, there are holding cost for the cash reserve if it is positive, but also interest (cost) in case it is negative. The cash reserve can be increased or decreased by the management at each decision epoch which implies transfer costs. What is the optimal cash balance policy?

• Mean-Variance Problem
  Consider a small investor who acts on a given financial market. Her aim is to choose among all portfolios which yield at least a certain expected return (benchmark) after periods, the one with smallest portfolio variance. What is the optimal investment strategy?

• Dividend Problem in Risk Theory
  Imagine we consider the risk reserve of an insurance company which earns some premia on the one hand but has to pay out possible claims on the other hand. At the beginning of each period the insurer can decide upon paying a dividend. A dividend can only be paid when the risk reserve at that time point is positive. Once the risk reserve got negative we say that the company is ruined and has to stop its business. Which dividend pay-out policy maximizes the expected discounted dividends until ruin?

• Bandit Problem
  Suppose we have two slot machines with unknown success probability and . At each stage we have to choose one of the arms. We receive one Euro if the arm wins, else no cash flow appears. How should we play in order to maximize our expected total reward over trials?

• Pricing of American Options
  In order to find the fair price of an American option and its optimal exercise time, one has to solve an optimal stopping problem. In contrast to a European option the buyer of an American option can choose to exercise any time up to and including the expiration time. Such an optimal stopping problem can be solved in the framework of Markov Decision Processes.

A Reference Book

You may buy it via Amazon, or find it via Springer Link

Markov Decision Models

Defining A Markov Decision Models

Markov Decision Model with planning horizon consists of a set of data with the following meaning for :

• is the state space, endowed with a -algebra . The elements (states) are denoted by

• is the action space, endowed with a -algebra . The elements (actions) are denoted by

• is a measurable subset of and denotes the set of possible state-action combinations at time . We assume that contains the graph of a measurable mapping , i.e. for all . For ,the set is the set of admissible actions in state at time .

• is a stochastic transition kernel from to , i.e. for any fixed pair , the mapping is a probability measure on and is measurable for all .The quantity gives the probability that the next state at time is in if the current state is and action a is taken at time . describes the transition law.

• is a measurable function. gives the (discounted) one-stage reward of the system at time if the current state is and action a is taken.

• is a measurable mapping. gives the (discounted) terminal reward of the system at time if the state is .

An equivalent definition of MDP

A Markov Decision Model is equivalently described by the set of data with the following meaning:

• are as in Definition of last slide

• is the disturbance space, endowed with a -algebra .

• is a stochastic transition kernel for and and denotes the probability that is in if the current state is and action is taken.

• is a measurable function and is called transition or system function. gives the next state of the system at time if at time the system is in state ,action is taken and the disturbance occurs at time .

Example: Consumption Problem

• Decision rule & strategy

  • A measurable mapping with the property for all , is called a decision rule at time . We denote by the set of all decision rules at time .

  • A sequence of decision rules with is called an -stage policy or -stage strategy.

• Value function:

• Theorem: For it holds:

Finite Horizon Markov Decision Models

Finite Horizon Markov Decision Models

Integrability Assumption (): For

Assumption () is assumed to hold for the -stage Markov Decision Problems.

Example: (Consumption Problem) In the consumption problem Assumption () is satisfied if we assume that the utility functions are increasing and concave and for all , because then and can be bounded by an affine-linear function with and since a.s. under every policy, the function satisfies

For and a policy let be defined by

is the expected total reward at time over the remaining stages to if we use policy and start in state at time . The value function is defined by

is the maximal expected total reward at time over the remaining stages to if we start in state at time .

The functions and are well-defined since

The Bellman Equation

Let us denote by , we define the following operators:

• For define

whenever the integral exists.

• For and define

• For define

is called the maximal reward operator at time .

Reward Iteration

Theorem
Let be an -stage policy. For it holds:

• and ,
• .

Example: (Consumption Problem) Note that for the operator in this example reads

Now let us assume that for all and . Moreover, we assume that the return distribution is independent of and has finite expectation . Then () is satisfied as we have shown before. If we choose the -stage policy with and , i.e. we always consume a constant fraction of the wealth, then the Reward Iteration implies by induction on that

Hence, with and maximizes the expected log-utility (among all linear consumption policies).

Maximizer, the Bellman Equation & Verification Theorem

• Definition of a maximizer: Let . A decision is called a maximizer of at time if , i.e. for all , is a maximum point of the mapping ,

• The Bellman Equation

• Verification Theorem: Let be a solution of the Bellman equation. Then it holds:

  • for .

  • If is a maximizer of for , then and the policy is optimal for the -stage Markov Decision Problem.

The Structure Assumption & Structure Theorem

• Structure Assumption (): There exists sets and such that for all :

  • If then is well-defined and

  • For all there exists a maximizer of with

• Structure Theorem: Let () be satisfied. Then it holds:
  

  • and the sequence () satisfies the Bellman equation, i.e. for

• A corollary: Let () be satisfied. If $ n \leq m \leq N$ then it holds:

• Backward Induction Algorithm

• Principle of Dynamic Programming: Let () be satisfied, Then it holds for :

i.e. if is optimal for the time period then is optimal for .

The Financial Markets

Asset Dynamics and Portfolio Strategies

• We assume that asset prices are monitored in discrete time

  • time is divided into periods of length and
  • multiplicative model for asset prices:
  • The binomial model(Cox-Ross-Rubinstein model) and discretization of the Black-Scholes-Merton model are two important special cases of the multiplicative model

• An N-period financial market with risky assets and one riskless bond

  • A riskless bond with and
  • There are risky assets and the price process of asset is given by and

• A portfolio or a trading strategy is an ()-adapted stochastic process where and for . The quantity denotes the amount of money which is invested into asset during the time interval .

• The vector is called the initial portfolio of the investor. The value of the initial portfolio is given by

where denotes the inner product of the vectors and .

• Let be a portfolio strategy and denote by the value of the portfolio at time before trading. Then

• The value of the portfolio at time after trading is given by

• Self-financing: A portfolio strategy is called self-financing if

for all , i.e. the current wealth is just reassigned to the assets.

• Arbitrage opportunity: An arbitrage opportunity is a self-financing portfolio strategy with the following property: and

• A theorem: Consider an -period financial market. The following two statements are equivalent:

  • There are no arbitrage opportunities.
  • For and for all -measurable it holds:

Modeling and Solution Approaches with MDP

• Modeling approach: specify the elements of the MDP

  • state space , action space
  • transition function:
  • value function
  • the Bellman equation:

• Solution approaches

  • backward induction (with the Bellman equation)
  • the existence and form of optimal policy
  • we are interested in the structured properties that are preserved in the iterations

MDP Applications in Finance

A Cash Balance Problem

• The cost have to be paid at the beginning of a period for cash level
  • , ,
  • is convex

Problem Formulation

• Elements of MDP

  • where denotes the cash level,
  • where denotes the new cash level after transfer,
  • ,
  • where denotes the cash change,
  • ,
  • ,
  • ,

• The state transition function:

• The value function

• The Bellman equation

Solution