Regularization for Deep Nets (Dropout & Weight Decay)

Weight decay, dropout, and early stopping against overfitting.

The idea

Regularization is any change to a training procedure that accepts a worse fit to the training data in exchange for a smaller gap between training and test error. It answers overfitting: a network with enough parameters can drive its training loss to nearly nothing by fitting the particular sample it was handed, at which point test error stops following training error down and begins to rise. Three regularizers are standard for deep networks.

Weight decay adds an $L^{2}$ penalty to the objective: $\tilde{J}(w) = J(w) + \frac{\alpha}{2}\lVert w \rVert_{2}^{2}.$ Large weights now cost something, so the fit keeps a weight large only where doing so reduces the data term by more. The penalty is differentiable, so it contributes its own term to every gradient step alongside the data term.

Dropout deletes each unit independently, with probability $1 - p$, on every training step. Each step therefore trains a different thinned network, and all of them share the surviving weights, so no unit can rely on any particular other unit being present. At test time nothing is dropped, so we must rescale the activations by $p$ somewhere for the two regimes to agree.

Early stopping halts training once validation error stops improving and keeps the parameters from that point rather than the last one.

Ways to work on it

• Walkthrough. How weight decay reshapes the gradient update and how dropout trains an ensemble of thinned networks.
• Practice. A weight-decay update or a dropout expectation.
• Hardest. Derive the closed-form shrinkage of L^2 weight decay at a quadratic optimum.

Not sure where to start? Take the ten-question placement test.