Cross-Entropy Loss

L = - p — punish confident, wrong predictions.

The idea

Cross-entropy is the standard loss for classifiers that output probabilities: it scores a prediction by the probability the model placed on the label that actually occurred. For a binary label $y \in \{0, 1\}$ and a predicted probability $p$ that $y = 1$, the loss on one example is $L = -\big[y\ln p + (1-y)\ln(1-p)\big].$ Only one term survives: when $y = 1$ the loss is $-\ln p$, and when $y = 0$ it is $-\ln(1-p)$. Either way $L$ is $-\ln$ of the probability the model gave to the truth.

The logarithm sets the penalties. Full confidence in the correct outcome costs $-\ln 1 = 0$; smaller probabilities cost more; and the cost grows without bound as the probability given to the truth approaches $0$. Squared error $(1-p)^{2}$, by contrast, charges at most $1$ for being certain and wrong, so it presses only weakly against a confident mistake.

With more than two classes the model emits one score $z_{i}$ per class, and the softmax $\mathrm{softmax}(z)_{i} = \frac{e^{z_{i}}}{\sum_{j} e^{z_{j}}}$ turns the scores into probabilities: exponentials are positive, and dividing by their total makes the outputs sum to $1$.

Ways to work on it

• Walkthrough. Binary cross-entropy, why it beats squared error, and softmax.
• Practice. Read off the softmax prediction from logits.
• Hardest. Probe how softmax behaves and connect cross-entropy to likelihood.

Not sure where to start? Take the ten-question placement test.