Reinforcement Learning for Finance

Markov Decision Process (MDP) — five components

An MDP is the tuple (S, A, P, R, γ): states S, actions A, transition kernel P(s'|s,a), reward function R(s,a), and disco

Return Gₜ (discounted)

Gₜ = Rₜ₊₁ + γ Rₜ₊₂ + γ² Rₜ₊₃ + … = Σₖ₌₀^∞ γᵏ Rₜ₊ₖ₊₁. The return is the discounted sum of future rewards starting from ti

State-value function Vπ(s)

Vπ(s) = 𝔼π[ Gₜ | Sₜ = s ] — the expected return when starting in state s and following policy π thereafter.

Action-value function Qπ(s,a)

Qπ(s,a) = 𝔼π[ Gₜ | Sₜ = s, Aₜ = a ] — expected return after taking action a in state s, then following π. Lets you comp

Markov property in MDPs

The next state and reward depend only on the present state–action pair: P(Sₜ₊₁, Rₜ₊₁ | S₀,A₀,…,Sₜ,Aₜ) = P(Sₜ₊₁, Rₜ₊₁ | S