Ridge & Lasso Regularization
L2 and L1 penalties that shrink coefficients and tame overfitting.
The idea
Ridge and lasso regularization penalize the size of regression coefficients to prevent overfitting. Least squares chooses the coefficients $\beta$ minimizing the residual sum of squares $\mathrm{RSS}(\beta)$ and asks nothing else of them; with many features, or features that nearly duplicate one another, those coefficients come out large and unstable, big positive and negative values offsetting each other to fit the particular sample.
Regularization adds a penalty, so that size itself costs something: $\mathrm{RSS}(\beta) + \lambda \sum_{j=1}^{p} \beta_{j}^{2} \qquad \mathrm{RSS}(\beta) + \lambda \sum_{j=1}^{p} |\beta_{j}|$ The first is ridge, penalizing squared coefficients; the second is lasso, penalizing absolute values. In both, $\lambda \ge 0$ sets the exchange rate between fitting the data and keeping $\beta$ small: at $\lambda = 0$ the penalty vanishes and the fit is ordinary least squares, and as $\lambda$ grows the penalty pulls the coefficients toward $0$.
The two penalties pull differently. The squared penalty pulls on a coefficient in proportion to its size, so the pull weakens as the coefficient nears $0$, and ridge coefficients become small without reaching $0$. The absolute-value penalty pulls with the same strength however small the coefficient, so lasso sets any coefficient that reduces $\mathrm{RSS}$ too little exactly to $0$, and its feature leaves the model.
The figure shows why, writing each penalty as a constraint on a two-coefficient model $x = (x_{1}, x_{2})$: the circular level sets of $\mathrm{RSS}$ grow out from the least-squares fit $\hat{x}$ until they first touch the allowed region, and the $\ell_{1}$ diamond $\|x\|_{1} \le t$ is touched at a corner, where one coordinate is exactly $0$, while the $\ell_{2}$ disk $\|x\|_{2} \le t$ is touched at a point off both axes.
Ways to work on it
- Walkthrough. Why regularize, ridge versus lasso, and a worked ridge estimate.
- Practice. Match a behavior to its penalty type or choice of lambda.
- Hardest. Derive the lasso soft-thresholding estimate and contrast it with ridge.
Not sure where to start? Take the ten-question placement test.