Differential Entropy

The continuous analog of entropy: h(X) = -∫ f f.

The idea

Differential entropy carries the entropy of a discrete random variable over to a continuous one.

A continuous variable offers no outcomes to sum over, since each individual value has probability zero, so we replace the probability mass function by the density $f$ and the sum by an integral.

Definition (Differential entropy).

Let $X$ be a continuous random variable with density $f$. The differential entropy of $X$ is $h(X) = -\int f(x)\,\ln f(x)\,dx,$ measured in nats with the natural logarithm, or in bits with $\log_2$ in its place.

The formula mirrors the discrete one, but the quantity behaves differently. A probability satisfies $p \le 1$, so every discrete term $-p \log p$ is non-negative and $H \ge 0$ always. A density obeys no such bound: wherever $f(x) > 1$ the integrand $-f \ln f$ is negative, and $h(X)$ itself can be negative. The uniform density on an interval of width $a$ makes this concrete: its height is $f = 1/a$ — the area under it stays $1$ — and that height exceeds $1$ exactly when $a < 1$. The reason is that $h$ measures how spread out the density is relative to a fixed unit of length rather than counting anything; measure $X$ in different units and $h(X)$ changes, which no genuine count of bits could do.

Ways to work on it

• Walkthrough. Define h(X), compute it for a uniform, and see why it can be negative.
• Practice. Find the exact differential entropy of a uniform interval.
• Hardest. The scaling law and the maximum-entropy Gaussian.

Not sure where to start? Take the ten-question placement test.