Maximum Entropy Distributions

Moment constraints pick the entropy-maximizing exponential family.

The idea

The maximum entropy principle selects a distribution when only a few of its averages are known: among all distributions satisfying the constraints, take the one of largest entropy. This is the least committed choice — any other candidate asserts structure the constraints do not contain.

Formally, maximize the differential entropy $h(p) = -\int p(x)\ln p(x)\,dx$, measured in nats since the logarithm is natural, over densities $p$, subject to $\int p\,dx = 1$ and to moment constraints $\mathbb{E}[f_i(X)] = \alpha_i$. Attach a Lagrange multiplier to each constraint and set the variational derivative to zero. The solution always takes one form:

$p(x) = \exp\Bigl(\lambda_0 + \sum_i \lambda_i f_i(x)\Bigr),$

an exponential family, with the multipliers $\lambda_i$ then fixed by requiring the constraints to hold.

The constraint functions $f_i$ appear directly in the exponent, so which averages are fixed decides the answer: no constraint gives the uniform distribution, a fixed mean on $[0, \infty)$ gives the exponential distribution, and a fixed variance on $\mathbb{R}$ gives the Gaussian. Finding the distribution reduces to solving for a few multipliers rather than searching over all densities.

Ways to work on it

• Walkthrough. The variational principle and the uniform, exponential, and Gaussian cases.
• Practice. Match a constraint set to its maximum entropy distribution.
• Hardest. Derive a maximum entropy distribution from its constraints.

Not sure where to start? Take the ten-question placement test.