Linear Regression

Fit y = wx + b by minimizing squared error.

The idea

Linear regression predicts an output $y$ from an input $x$ by fitting a straight line to observed pairs: $\hat{y} = wx + b,$ where the slope $w$ scales the input and the intercept $b$ shifts the result. Choosing the line means choosing those two numbers.

Unless the points happen to lie on a line, no choice predicts every $y$ exactly, so we need a measure of how well a line fits. Each observation has a residual $y_{i} - \hat{y}_{i}$, the amount the line misses it by. The sum of the residuals is a poor measure, since misses above and below the line cancel. Squaring each residual removes the signs, and the standard measure is the mean of the squares, $\mathrm{MSE} = \frac{1}{n}\sum_{i}(y_{i} - \hat{y}_{i})^{2}.$ Squaring also counts a miss of $4$ as sixteen times a miss of $1$, so the best line accepts several small errors to avoid one large one.

The $w$ and $b$ minimizing the mean squared error give the least-squares line. The error is a quadratic in $w$ and $b$, so we find them exactly by setting its two derivatives to zero and solving.

Ways to work on it

• Walkthrough. The model, its slope and intercept, and the squared-error objective.
• Practice. Recover the slope of a regression line from two points.
• Hardest. Apply the least-squares closed form to noisy data.

Not sure where to start? Take the ten-question placement test.