Vanishing Gradients
Vanishing gradients are covered in both ML-10-DL and ML-TS > Things to be aware of.
Gradients shrink as backpropagation moves toward the lower layers (or back through earlier time steps in an RNN). The early layers' weights barely change, so training stalls or settles on a poor solution.
Weight update: \(w \leftarrow w - \eta \cdot \frac{\partial L}{\partial w}\). A tiny gradient means a tiny update.
The opposite problem is exploding gradients: gradients grow layer by layer until training diverges. It's common in RNNs.
Why it happens¶
-
By the chain rule, a lower layer's gradient is the product of many per-layer derivatives. If those factors are mostly < 1, the product shrinks exponentially with depth (or with sequence length, in RNNs).
-
Saturating activations: the sigmoid's derivative is at most 0.25, and close to 0 for large |z|.
-
Initialization: the old default (normal init with std 1, plus sigmoid) made each layer's output variance larger than its input variance, pushing the top layers into saturation (Glorot & Bengio, 2010).
Fixes¶
| Fix | How it helps |
|---|---|
| ReLU (hidden layers) | Doesn't saturate for positive inputs |
| Leaky ReLU / PReLU / ELU | Fix dying ReLUs (neurons stuck outputting 0, with zero gradient) via a small, learned (PReLU), or smooth (ELU) negative slope |
| Glorot / He initialization | Keeps output variance ≈ input variance from layer to layer. Glorot for sigmoid/tanh/softmax; He for ReLU and its variants |
| Batch normalization | Normalizes each layer's inputs per mini-batch, then learns a scale and shift |
| Gradient clipping | Caps gradients at a threshold. Targets exploding gradients, especially in RNNs |
| LSTM / GRU cells | Gates and a cell state let information and gradients flow across many time steps |
| Seq-to-seq training | Computes a loss at every time step (not just the last), so more gradient signal reaches the early steps |