Lecture 04

Reinforcement Learning (RL) is a branch of machine learning that enables agents to learn optimal actions based on rewards from the environment.

RL learns from interaction with the environment Goal: learn policy to maximize expected rewards Core elements: states, actions, rewards, transitions, discount factor

• Distinct from supervised learning, RL is driven by exploration and exploitation strategies.

Reinforcement learning mimics decision-making processes in real-world scenarios, making it applicable in finance.

• Algorithmic Trading: Develop strategies to buy/sell assets based on predicted market movements.

• Portfolio Optimization: Automatically adjust asset allocations to achieve desired return profiles.

• Risk Management: Create adaptive systems to monitor and mitigate financial risks dynamically.

• Mathematical Formulation (Maximize expected return)

The practical use of RL showcases its capabilities in addressing financial challenges.

• Classical Dynamic Programming (DP) established the foundation for sequential decision-making under uncertainty.

  In finance, DP solves problems such as portfolio optimization, option pricing, or consumption–investment planning — but these models require a known transition structure and suffer from the curse of dimensionality.

• Reinforcement Learning (RL) removes the reliance on explicit models.

  By learning from interactions or simulations, RL estimates value functions and policies directly, enabling data-driven control for trading, execution, and risk management tasks.

• Deep Reinforcement Learning (Deep RL) merges neural networks with RL to approximate complex value or policy functions, scaling to high-dimensional features like historical returns, order-book data, or textual sentiment.

  This evolution—from theory-driven DP to data-driven Deep RL—allows automated agents to operate effectively in realistic, uncertain financial markets.

Each stage advances our capacity to represent complexity and uncertainty in financial systems:

• DP: mathematically exact, interpretable, but computationally intractable for high-dimensional markets.
• RL: model-free and flexible, learns directly from data but may require extensive exploration and careful reward design.
• Deep RL: scalable and expressive, capturing nonlinear dependencies among financial variables, though prone to instability and overfitting.

Together, these methods reveal a clear trajectory — from model-based optimization to experience-based learning. This progression provides practical tools to tackle portfolio allocation, dynamic hedging, and algorithmic trading in environments where traditional models no longer suffice.

• Core Components:
  • Agent: Learner making decisions.
  • Environment: External factors affecting the agent.
  • State (): Current situation of the agent.
  • Action (): Decisions made by the agent.
  • Reward (): Feedback received from the environment.

Understanding these elements is crucial for applying RL techniques effectively in finance.

• State Space Design:

  • Incorporate essential financial metrics such as asset prices, volatility, and indicators.

• Action Space Design:

  • Define potential actions, e.g., buy, sell, or hold.
  • Specify trading quantities or rebalancing strategies.

Careful design of these spaces is critical for the effectiveness of RL agents in finance.

• The reward function drives the learning process of the agent by evaluating the effectiveness of its actions.

  • Common formulations include return on investment and risk-adjusted returns.

• A well-designed reward structure can guide the agents toward long-term profitability.

The formulation of the reward function is essential to align the agent's behavior with financial objectives.

• denotes the state space of the system. A state is the information which is available for the controller at time . Given this information an action has to be selected.

• denotes the action space. Given a specific state at time , a certain subclass of actions may only be admissible.

• is a stochastic transition kernel which gives the probability that the next state at time is in the set B if the current state is and action a is taken at time .

• gives the (discounted) one-stage reward of the system at time if the current state is and action a is taken

• gives the (discounted) terminal reward of the system at the end of the planning horizon.

• A control is a sequence of decision rules () with where determines for each possible state the next action at time . Such a sequence is called policy or strategy.
• Formally the Markov Decision Problem is given by

• Dividend Problem in Risk Theory: Imagine we consider the risk reserve of an insurance company which earns some premia on the one hand but has to pay out possible claims on the other hand. At the beginning of each period the insurer can decide upon paying a dividend. A dividend can only be paid when the risk reserve at that time point is positive. Once the risk reserve got negative we say that the company is ruined and has to stop its business. Which dividend pay-out policy maximizes the expected discounted dividends until ruin?

• Bandit Problem: Suppose we have two slot machines with unknown success probability and . At each stage we have to choose one of the arms. We receive one Euro if the arm wins, else no cash flow appears. How should we play in order to maximize our expected total reward over trials?

• Pricing of American Options
  In order to find the fair price of an American option and its optimal exercise time, one has to solve an optimal stopping problem. In contrast to a European option the buyer of an American option can choose to exercise any time up to and including the expiration time. Such an optimal stopping problem can be solved in the framework of Markov Decision Processes.

A Markov Decision Model is equivalently described by the set of data with the following meaning:

• are as in Definition of last slide

• is the disturbance space, endowed with a -algebra .

• is a stochastic transition kernel for and and denotes the probability that is in if the current state is and action is taken.

• is a measurable function and is called transition or system function. gives the next state of the system at time if at time the system is in state ,action is taken and the disturbance occurs at time .

We denote by the random return of our risky asset over period . Further we suppose that are non-negative, independent random variables and we assume that the consumption is evaluated by utility functions . The final capital is also evaluated by a utility function . Thus we choose the following data:

• where denotes the wealth of the investor at time
• where denotes the wealth which is consumed at time
• for all , i.e. we are not allowed to borrow money
• where denotes the random return of the asset
• is the transition function
• distribution of (independent of )
• is the one-stage reward

• Decision rule & strategy
  • A measurable mapping with the property for all , is called a decision rule at time . We denote by the set of all decision rules at time .
  • A sequence of decision rules with is called an -stage policy or -stage strategy.
• Value function:

• Theorem: For it holds:

Integrability Assumption (): For

Assumption () is assumed to hold for the -stage Markov Decision Problems.

Example: (Consumption Problem) In the consumption problem Assumption () is satisfied if we assume that the utility functions are increasing and concave and for all , because then and can be bounded by an affine-linear function with and since a.s. under every policy, the function satisfies

• is the expected total reward at time over the remaining stages to if we use policy and start in state at time .

• The value function is the maximal expected total reward at time over the remaining stages to if we start in state at time .

• The functions and are well-defined since:

Let us denote by , we define the following operators:

• : for define (whenever the integral exists)

• : for and define

• : for define (the maximal reward operator at time .)

Theorem (Reward Iteration): Let be an -stage policy. For it holds:

• and ,
• .

Example: (Consumption Problem) Note that for the operator in this example reads

Now let us assume that for all and . Moreover, we assume that the return distribution is independent of and has finite expectation . Then () is satisfied as we have shown before. If we choose the -stage policy with and , i.e. we always consume a constant fraction of the wealth, then the Reward Iteration implies by induction on that

Hence, with and maximizes the expected log-utility (among all linear consumption policies).

• Definition of a maximizer: Let . A decision is called a maximizer of at time if , i.e. for all , is a maximum point of the mapping ,

• The Bellman Equation

• Verification Theorem: Let be a solution of the Bellman equation. Then it holds:

  • for
  • If is a maximizer of for , then and the policy is optimal for the -stage Markov Decision Problem.

• Principle of Dynamic Programming: Let () be satisfied, Then it holds for :
  i.e. if is optimal for the time period then is optimal for .

• Asset Dynamics and Portfolio Strategies: We assume that asset prices are monitored in discrete time

  • time is divided into periods of length and
  • multiplicative model for asset prices:
  • The binomial model(Cox-Ross-Rubinstein model) and discretization of the Black-Scholes-Merton model are two important special cases of the multiplicative model

• An N-period financial market with risky assets and one riskless bond

  • A riskless bond with and
  • There are risky assets and the price process of asset is given by and

• A portfolio or a trading strategy is an ()-adapted stochastic process where and for . The quantity denotes the amount of money which is invested into asset during the time interval .
• The vector is called the initial portfolio of the investor. The value of the initial portfolio is given by

• Let be a portfolio strategy and denote by the value of the portfolio at time before trading. Then

• The value of the portfolio at time after trading is given by

• Self-financing: A portfolio strategy is called self-financing if

  for all , i.e. the current wealth is just reassigned to the assets.

• Arbitrage opportunity: An arbitrage opportunity is a self-financing portfolio strategy with the following property: and

• A theorem: Consider an -period financial market. The following two statements are equivalent:

  • There are no arbitrage opportunities.
  • For and for all -measurable it holds:

• Modeling approach: specify the elements of the MDP

  • state space , action space
  • transition function:
  • value function
  • the Bellman equation:

• Solution approaches

  • backward induction (with the Bellman equation)
  • the existence and form of optimal policy
  • we are interested in the structured properties that are preserved in the iterations

• involves the decision about the optimal cash level of a firm over a finite number of periods, a random stochastic change in the cash reserve each period (which can be both positive and negative).
• costs
  • holding cost or opportunity cost for the cash reserve if it is positive
  • out-of-pocket expense
• The cash reserve can be increased or decreased by the management at the beginning of each period
  • transfer costs

  • random changes in the cash flow are given by independent and identically distributed random variables () with finite expectation
• The cost have to be paid at the beginning of a period for cash level :

• Elements of MDP
  • where denotes the cash level,
  • where denotes the new cash level after transfer,
  • ,
  • where denotes the cash change,
  • ,
  • ,
  • ,

• The state transition function:

• The value function

• The Bellman equation

• Solution is worked through by backward induction
• We verified the validity of Integrability Assumption () and the relative Structure Assumption () for each
• The solution to the cash balance problem (Theorem 2.6.2):
  • There exist critical levels and such that for
    with .
  • There exist critical levels and such that for