Advances of RL in Finance

Hambly, B., Xu, R., & Yang, H. (2023). Recent advances in reinforcement learning in finance. arXiv preprint arXiv:2112.04553.

The Basics of Reinforcement Learning

Setup: Markov Decision Processes

• Infinite Time Horizon and Discounted Rewards
  • state space
  • action space
  • the value function for each

subject to

• • the Dynamic Programming Principle (DPP)

We write the value function as

• the -function

• the Bellman equation for the -function

• Finite Time Horizon

  • the value function

subject to

• the Dynamic Programming Principle (DPP)

• the terminal condition

We write the value function as

• • the -function

• the Bellman equation for the -function

• the terminal condition

for all .

Theorem Assume and with probalbility one. For any infinite horizon discounted MDP, there always exists a deterministic stationary policy that is optimal.

• Linear MDPs and Linear Functional Approximation
  • the infinite horizon setting, an MDP is said to be linear with a feature map , if there exist unknown (signed) measures over and an unknown vector such that for any , we have

• the finite horizon setting, an MDP is said to be linear with a feature map , if for any , there exist unknown (signed) measures over and an unknown vector such that for any , we have

Typically fcatures are assumed to be known to the agent and bounded, namely, for all

• Nonlinear Functional Approximation
  • nonlinear functional approximations do not require knowledge of the kernel functions a priori
  • The most popular nonlincar functional approximation approach is to use neural networks (the universal approximation theorem)
  • gradient-based algorithms for certain neural network architectures enjoy provable convergence guarantees

From MDP to Learning

• RL Problem
  • the transition dynamics and the reward function for the infinite horizon MDP problem are unknown
  • to find an optimal stationary policy (if it exists) while simultaneously learning the unknown and
  • The learning of and can be either explicit or implicit
    • model-based RL
    • model-free RL

• Agent-environment Interface
  • The learner or the docision-maker is called the agent
  • The physical world that the agent operates in and interacts with
  • The agent and environment interact at each of a sequence of discrete time steps, , in the following way.
    • At the beginning of each time step , the agent receives some representation of the environment's states, and selects an action .
    • At the end of this time step, the agent receives a numerical reward (possibly stochastic) and a new state from the environment.
  • The tuple is called a sample at time
  • is referred to as the history or experience up to time t
  • An RL algorithm is a finite sequence of well-defined policies for the agent to interact with the environment.

• Exploration vs Exploitation
  • the agent learns to select her actions based on her past experiences (exploitation) and/or by trying new choices (exploration).
    • Exploration provides opportunities to improve performance from the current sub-optimal solution to the ultimate globally optimal one, yet it is time consuming and computationally expensive as over-exploration may impair the convergence to the optimal solution.
    • Pure exploitation, tends to yield sub-optimal global solutions.
  • An appropriate trade-olf between exploration and exploitation is crucial in the design of RL algorithms in order to improve the learning and the optimization performance.

• Simulator

  • a.k.a. generative model
  • allows the algorithm to query arbitrary state-action pairs and return the reward and the next state
  • the agent can "restart" the system at any time
  • it significantly alleviates the difficulty of exploration

• Randomized Policy vs Deterministic Policy

  • A randomized policy is also known as a relaxed control (in the control literature) and as a mixed strategy (in game theory).
  • most of the RL algorithms adopt randomized policies to encourage exploration when agents are not certain about the environment.

Performance Evaluation

• episodic criterion
  • (usually) a finite time horizon
  • one episode contains a sequence of states, actions and rewards, which starts at time 0 and ends at the terminal time
  • the performance is evaluated in terms of the total number of samples in all episodes
  • the episodic criterion can also be used to analyze infinite horizon problems

• Notations
  • • the order of the scaling in and (when these are finite)
    • the discount rate (in the infinite horizon case)
    • the dimension of the features （when functional approximation is used）
  • • only include the dependence on
  • • a polynomial function of its arguments

Sample Complexity

• Sample complexity: the total number of samples required to find an approximately optimal policy

  • Sample:
  • can be used for any kind of RL problem

• Sample complexity for episodic RL

  • the minimum number of samples such that for all is -optimal with probability at least , i.e., for ,

• Sample complexity for discounted RL
  • the -sample complexity:
    • the minimum number of samples such that for all is -close to , that is

• • • it holds either with high probability (that is ) or in the expectation sense

• • the -sample complexity:
    • the minimum number of samples such that for all is -close to , that is

• • • it holds either with high probability (in that or in the expectation sense

• • sample complexity of exploration
    • the number of samples such that the non-stationary policy at time is not -optimal for the current state
    • it counts the number of mistakes along the whole trajectory
    • is the minimum number of samples such that for all is -optimal when starting from the current state with probability at least , i.e., for ,

• • the mean-square sample complexity
    • measures the sample complexity in the average sense with respect to the steady state distribution or the initial distribution
    • is defined as the minimum number of samples such that for all ,

Rate of Convergence

• uses the relationship between the number of iterations/steps and the error term , to quantify how fast the learned policy converges to the optimal solution
• is also called the iteration complexity
• it calculates the number of iterations needed to reach a given accuracy while ignoring the number of samples used within each iteration.
• the rate of convergence coincides with the sample complexity when only one sample is used in each iteration
• the rate of convergence and the sample complexity are of the same order with respect to when a constant number of samples are used within each iteration

• different rate of converge
  • converge at a linear rate:
  • converge at a sublinear rate (slower than linear): with
  • converge at a supertinear (faster than linear):

Regret Analysis

• the difference between the cumulative reward of the optimal policy and that gathered by
• it quantifies the exploration-exploitation trade-off
• the regret of an algorithm after episodes with steps is

• currently no regret analysis for RL problems with infinite time horizon and discounted reward

Asymptotic Convergence

• requires the error term to decay to zero as goes to infinity (without specifying the order of )
• often a first step in the theoretical analysis of RL algorithms.

Classification of RL Algorithms

• Components of An RL Algorithm
  • model-free RL
    • a representation of a value function that provides a prediction of how good each state or each state/action pair is,
    • a direct representation of the policy or ,
  • model-based RL
    • a model of the environment (the estimated transition function and the estimated reward function) in conjunction with a planning algorithm (any computational process that uses a model to create or improve a policy).

• model-based algorithms
  • maintain an approximate MDP model by estimating the transition probabilities and the reward function
  • derive a value function from the approximate MDP
  • the policy is then derived from the value function
  • another line of model-based algorithms make structural assumptions on the model, using some prior knowledge, and utilize the structural information for algorithm design

• model-free algorithms
  • directly learn a value (or state-value) function or the optimal policy without inferring the model
  • model-free algorithms can be further divided into two categories
    • value-based methods: store only a value function without an explicit policy during the learning process
    • policy-based methods: explicitly build a representation of a policy and keep it in memory during learning

Value-based Methods

• setting
  • infinite time horizon with discounting
  • finite state and action spaces and
  • stationary policies

Temporal-Difference Learning (时序差分学习)

The agent can update her estimate of the value function at the iteration by

The TD update can also be written as

Q-learning Algorithm

• -learning is a stochastic approximation to the Bellman equation for the -function with samples observed by the agent
• At iteration , the -function is updated using a sample where ,

Theorem (Watkins and Dayan (1992)). Assume and . Let be the index of the -th time that the action is used in state s. Let be a constant. Given bounded rewards , learning rate and

then as with probability 1

SARSA

SARSA adopts a policy which is based on the agent's current estimate of the -function.

On-policy Learning vs. Off-policy Learning

• An off-policy agent learns the value of the optimal policy independently of the agent's actions.
• An on-policy agent learns the value of the policy being carried out by the agent including the exploration steps.

Policy-based Methods

• parametrization
  • the policy by a vector
  • : the probability distribution parameterized by over the action space given the state at time
  • the policy objective function for RL with infinite time horizon

• • the policy objective function for RL with finite time horizon

• the policy parameter is updated using the gradient ascent rule to maximize :

Policy Gradient Theorem

Theorem (Policy Gradient Theorem). Assume that is differentiable with respect to and that there exists , the stationary distribution of the dynamics under policy , which is independent of the initial state . Then the policy gradient is

REINFORCE: Monte Carlo Policy Gradient

Actor-Critic Methods