Bias–Variance Tradeoff

Error = bias² + variance + noise — balance simple vs complex.

The idea

The bias–variance decomposition explains where a model's test error comes from. Fit a model to a randomly drawn training set and measure its expected squared error on a fresh point. Averaged over the possible training sets, that error separates into three parts:

$\text{Error} = \text{Bias}^{2} + \text{Variance} + \text{Noise}.$

Bias is the gap between the truth and the model's average prediction over training sets. It measures what the model cannot represent: a straight line fitted to curved data misses in the same direction whichever sample it sees, so averaging does not remove the miss.

Variance measures how much the fitted model changes when the training set is redrawn. A model flexible enough to pass through every training point passes through different points next time, so its predictions swing from sample to sample.

Noise is randomness in the labels themselves. It belongs to the data, not the model, so no model removes it; it is a floor under the test error.

The first two terms trade against each other: more flexibility lets the model track the signal, which lowers bias, and also lets it track the sample's random scatter, which raises variance. Test error is smallest at a complexity between the two extremes.

Ways to work on it

• Walkthrough. Underfitting, overfitting, and the three-term decomposition.
• Practice. Diagnose a scenario as underfitting or overfitting.
• Hardest. Recover a missing term of the error decomposition and reason about regularization.

Not sure where to start? Take the ten-question placement test.