Deep Learning for Finance

A perceptron outputs y = φ(wᵀx + b), where w is a weight vector, b is a bias scalar, and φ is a (historically step) activation. A feedforward network stacks such units into layers with differentiable activations so gradients can flow.

Sigmoid σ(x) = 1/(1+e⁻ˣ) maps ℝ → (0,1); tanh(x) = (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) maps ℝ → (−1,1). Tanh is zero-centered, which usually trains faster than sigmoid; both saturate and cause vanishing gradients for large |x|.

ReLU(x) = max(0, x): cheap, non-saturating for x > 0, but neurons can 'die' when stuck at x < 0. Leaky ReLU(x) = x for x ≥ 0 and αx (e.g. α = 0.01) for x < 0, preserving a small gradient on the negative side.

GELU(x) = x · Φ(x), where Φ is the standard-normal CDF. It is a smooth, non-monotonic activation that approximates ReLU but lets small negative values pass with a tapered gradient. Standard in transformer architectures (BERT, GPT).

ELU(x) = x for x ≥ 0 and α(eˣ − 1) for x < 0, giving smooth, negative-side outputs that push mean activations toward zero. SELU multiplies ELU by a scale λ ≈ 1.0507 with α ≈ 1.6733; under specific weight init it gives self-normalizing networks where activation mean/variance stays at 0/1 through depth.

Swish(x) = x · σ(βx), often with β = 1 (SiLU). Smooth, non-monotonic, and self-gated by the sigmoid factor; empirically tends to beat ReLU on deep networks and is used inside EfficientNet, some transformers, and diffusion U-Nets.