Reinforcement Learning for Finance

An MDP is the tuple (S, A, P, R, γ): states S, actions A, transition kernel P(s'|s,a), reward function R(s,a), and discount factor γ ∈ [0,1). The Markov property requires that P and R depend only on the current (s,a).

Gₜ = Rₜ₊₁ + γ Rₜ₊₂ + γ² Rₜ₊₃ + … = Σₖ₌₀^∞ γᵏ Rₜ₊ₖ₊₁. The return is the discounted sum of future rewards starting from time t.

Vπ(s) = 𝔼π[ Gₜ | Sₜ = s ] — the expected return when starting in state s and following policy π thereafter.

Qπ(s,a) = 𝔼π[ Gₜ | Sₜ = s, Aₜ = a ] — expected return after taking action a in state s, then following π. Lets you compare actions without a model.

The next state and reward depend only on the present state–action pair: P(Sₜ₊₁, Rₜ₊₁ | S₀,A₀,…,Sₜ,Aₜ) = P(Sₜ₊₁, Rₜ₊₁ | Sₜ, Aₜ). When this fails (partial observability), use a POMDP or augment state with history features.

Episodic: trajectories terminate at an absorbing state and split into finite-length episodes (e.g., one trading day). Continuing: no terminal state; γ < 1 keeps Gₜ finite. For undiscounted continuing tasks, use the average-reward formulation J(π) = lim (1/T)·𝔼[Σ Rₜ].