Covariance & Correlation

Joint variation, its sign, and the unit-free correlation in [-1, 1].

The idea

Covariance measures whether two random variables deviate from their means together. Multiply the deviations $X - \mathbb{E}[X]$ and $Y - \mathbb{E}[Y]$: the product is positive on outcomes where both variables land above their means or both land below, and negative where they land on opposite sides. Its average therefore records the direction of the association: $\operatorname{Cov}(X, Y) = \mathbb{E}\big[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])\big] = \mathbb{E}[XY] - \mathbb{E}[X]\,\mathbb{E}[Y].$ The second form follows by expanding the product and applying linearity, and it is usually the easier one to compute. Setting $Y = X$ recovers the variance.

The sign of a covariance is informative; the size is not. The quantity carries the units of $X$ times the units of $Y$, so measuring a height in centimetres rather than metres multiplies it by one hundred while the association stays the same. Dividing by the spread of each variable cancels the units: $\rho(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sqrt{\operatorname{Var}(X)\,\operatorname{Var}(Y)}}.$ The correlation $\rho$ is a pure number between $-1$ and $1$, and it reaches $\pm 1$ exactly when one variable is a linear function of the other. It detects linear association only: two variables can be strongly related and still have $\rho = 0$.

Ways to work on it

• Walkthrough. Covariance, the independence case, and the correlation coefficient.
• Practice. Compute a covariance from given expectations.
• Hardest. Bilinearity of covariance and scale-invariance of correlation.

Not sure where to start? Take the ten-question placement test.