Maximum Likelihood Estimation

Pick the parameters that make the observed data most probable.

The idea

Maximum likelihood estimation estimates an unknown parameter $\theta$ by choosing the value under which the observed data was most probable.

A model with parameter $\theta$ assigns a probability to every dataset we might observe. Once one dataset is in hand, read that expression the other way round, as a function of $\theta$ with the data held fixed. It is then called the likelihood, and for independent observations $x_1, \ldots, x_n$ it is $L(\theta) = \prod_{i=1}^{n} \mathbb{P}(x_i \mid \theta),$ a product because independence makes the joint probability the product of the individual ones. $L(\theta)$ is a probability of the data, not of $\theta$, and it does not add up to $1$ across values of $\theta$. The estimate $\hat{\theta}$ is the value of $\theta$ making $L(\theta)$ largest.

We maximize the log-likelihood $\ell(\theta) = \log L(\theta)$ rather than $L$ itself. This cannot change the answer, since $\log$ is increasing and so is largest wherever $L$ is, but it simplifies the algebra: the logarithm turns the product into a sum, which we can differentiate term by term. Setting $\ell'(\theta) = 0$ then locates the maximum.

Ways to work on it

• Walkthrough. Build the likelihood, take logs, and derive the Bernoulli MLE.
• Practice. Estimate a success probability as a sample proportion.
• Hardest. Derive least squares from Gaussian-noise maximum likelihood.

Not sure where to start? Take the ten-question placement test.