Logistic Regression

P(y=1) = (wx + b) — squash a score into a probability.

The idea

Logistic regression predicts the probability that a binary outcome $y$ equals $1$ from a linear score $z = wx + b$. The score can be any real number, and a probability must lie in $[0, 1]$, so the model passes the score through the sigmoid $\sigma(z) = \frac{1}{1 + e^{-z}}$ and predicts $\mathbb{P}(y = 1) = \sigma(wx + b)$.

Since $e^{-z}$ is positive for every real $z$, the denominator exceeds $1$, so $\sigma(z)$ lies strictly between $0$ and $1$. The function is increasing, approaches $0$ as $z$ runs to $-\infty$ and $1$ as $z$ runs to $+\infty$, and takes the value $\tfrac{1}{2}$ at $z = 0$.

Predicting the more probable class means predicting $y = 1$ exactly when $\sigma(z) > \tfrac{1}{2}$, and since $\sigma$ is increasing with $\sigma(0) = \tfrac{1}{2}$, that happens exactly when $z > 0$. The decision boundary is therefore where the linear score vanishes: the single input $x = -b/w$ with one feature, a line or a plane with more. Inverting the sigmoid gives $z = \ln\!\big(\tfrac{p}{1-p}\big)$, the logit, so the score is the log-odds of the positive class.

Ways to work on it

• Walkthrough. Turn a score into a probability with the sigmoid, and locate the decision boundary.
• Practice. Find the input where a model's decision boundary sits.
• Hardest. Push the sigmoid to its limits, invert it, and classify a point.

Not sure where to start? Take the ten-question placement test.