ROC Curves & AUC

Threshold sweeps, TPR vs FPR, and area under the curve.

The idea

The ROC curve and the AUC evaluate a binary classifier that outputs a score rather than a label. The score becomes a label only once we fix a threshold $t$: call an example positive when its score is at least $t$. Every choice of $t$ gives a different classifier from the same scores, so a single accuracy figure describes one arbitrary choice rather than the scores themselves.

Each threshold produces four counts — true and false positives, true and false negatives — which two rates summarize: $\text{TPR} = \frac{\text{TP}}{\text{TP} + \text{FN}}, \qquad \text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}.$ The first is the fraction of the actual positives the classifier catches, the second the fraction of the actual negatives it wrongly flags. Each rate is computed inside one true class, so neither changes when the classes become rarer or more common.

Lowering $t$ labels more examples positive, which can only push both rates up. Sweeping $t$ from high to low therefore traces a path from $(0,0)$ to $(1,1)$ in the plane whose axes are $\text{FPR}$ and $\text{TPR}$: the ROC curve. Scores that rank the two classes no better than chance trace the diagonal from $(0,0)$ to $(1,1)$. The better the scores rank positives above negatives, the further the curve bows toward the top-left corner, where the classifier catches positives before it starts flagging negatives.

The area under the curve is the AUC: one number for the whole sweep, depending only on the order of the scores and not on any threshold.

Ways to work on it

• Walkthrough. TPR, FPR, the ROC curve, and what AUC measures.
• Practice. Compute a rate from a confusion matrix at one threshold.
• Hardest. AUC as the probability a random positive outranks a random negative.

Not sure where to start? Take the ten-question placement test.