Skip to content

Over Under Fitting

Over/underfitting remedies appear in ML-01 > Algorithm Issues, ML-04-Regression, ML-07-Ensemble > The Bias/Variance Trade-off, and ML-10-DL > Tips for training NN.

A model overfits when it learns the noise in its training data and fails to generalize. It underfits when it's too simple to capture the real pattern.

Bias/variance

Image

Expected generalization error = bias² + variance + irreducible error

  • Bias: error from wrong assumptions (e.g., assuming linear when the relationship is quadratic). High bias → underfitting.

  • Variance: error from sensitivity to small variations in the training data. High variance → overfitting.

  • Irreducible error: noise in the data itself. Only better data (fixing sources, removing outliers) reduces it.

More model complexity → bias ↓, variance ↑. The goal is the sweet spot between the two.

Diagnosing

Overfitting (high variance) Underfitting (high bias)
Train vs validation error Train error low, validation error much higher Both high and close together
Learning curve (error vs training-set size) Persistent gap between the curves, narrowing as data grows Both curves plateau close together, at high error
Validation curve (score vs a complexity setting) Past the sweet spot, training score keeps rising while validation score falls Both scores low at low complexity
Will more data help? Usually No

Learning curve: Image. In sklearn: learning_curve, validation_curve.

Fixes

Overfitting Underfitting
Model Simpler model, fewer parameters More powerful model, more parameters
Features Fewer features (selection / reduction) Better features (feature engineering)
Constraints More Regularization Less regularization
Data More training data; reduce noise (fix errors, remove outliers) —
Training Early stopping (stop when validation error stops improving) Train longer

Model-specific levers

  • Decision trees: limit max_depth, raise min_samples_leaf, or prune.

  • Ensembles: bagging and random forests mainly reduce variance. Boosting mainly reduces bias but can overfit with too many estimators, so use shrinkage and early stopping. See Random-Subspaces-Patches.

  • Neural nets

    • Overfitting: Regularization (kernel_regularizer), early stopping, dropout (at each training step, every neuron except the outputs is dropped with probability p; nothing is dropped at inference).

    • Underfitting: more layers or neurons, a different activation function (e.g., maxout), a different optimizer or learning rate.