Regularization

One remedy for overfitting (see Over-Under-Fitting).

Regularization adds a penalty on the size of the weights to the loss: loss + α · penalty(w). The penalty always applies and grows with the weights. Larger α ⟶ simpler model ⟶ more bias, less variance.

• The bias/intercept term is usually not penalized.

• Scale features first. The penalty treats every weight equally, so unscaled features get penalized unevenly.

Strength hyperparameter

• sklearn Ridge / Lasso / ElasticNet: alpha (higher = stronger). ElasticNet's l1_ratio sets the L1/L2 mix (1 = pure Lasso).

• sklearn LogisticRegression / SVC: C is the inverse (smaller C = stronger).

• Keras: per-layer kernel_regularizer (l1, l2, or l1_l2).

Type	Penalty	Key Characteristics
L1 (Lasso)	Sum of absolute values of weights	- Feature selection: drives some weights to exactly 0 - Erratic with strongly correlated features (tends to keep one arbitrarily) - With gradient descent, bounces near the optimum ⟶ reduce the learning rate over time
L2 (Ridge)	Sum of squared values of weights	- Shrinks weights smoothly, rarely to 0 - More stable with correlated features - Good default
Elastic Net	Weighted mix of L1 and L2 (l1_ratio)	- Keeps L1's feature selection with L2's stability - Prefer over Lasso when features outnumber instances or are strongly correlated

Choosing: start with Ridge. Switch to Lasso or Elastic Net if you expect only a few features to matter.