Relative Entropy

Relative entropy compares one probability distribution with another. In probability and statistics it is usually called the Kullback–Leibler divergence, or KL divergence.

For two discrete probability distributions $P$ and $Q$ on the same outcome set, with probabilities $p_x$ and $q_x$, the relative entropy of $P$ with respect to $Q$ is

$$D(P\Vert Q) = \sum_x p_x\log\frac{p_x}{q_x}.$$

It is always nonnegative when it is well defined, and it is zero exactly when the two distributions agree. It is not symmetric, however, and it is not a metric.

Relative entropy is the canonical way to quantify how costly it is to model data drawn from $P$ as though they were drawn from $Q$. It also packages mutual information, likelihood-ratio evidence, statistical distinguishability, and the finite-dimensional quantum relative entropy preview used in quantum information.

Definition

Let $P$ and $Q$ be probability distributions on a finite or countable set $\Omega$. Write

$$p_x=P(\{x\}), \qquad q_x=Q(\{x\}).$$

The KL divergence is

$$D(P\Vert Q) = \sum_{x\in\Omega} p_x\log\frac{p_x}{q_x}.$$

The conventions are:

• if $p_x=0$, then the contribution is $0$;
• if $p_x>0$ and $q_x=0$, then $D(P\Vert Q)=+\infty$.

The support condition can be stated as

$$P\ll Q,$$

meaning that every event assigned probability zero by $Q$ is also assigned probability zero by $P$. If this absolute-continuity condition fails, relative entropy is infinite.

The logarithm base fixes the units. Natural logarithms give nats. Base-$2$ logarithms give bits.

Expected Log-Likelihood Ratio

Relative entropy is an expectation value under $P$:

$$D(P\Vert Q) = \mathbb E_P \left[ \log \frac{p_X(X)}{q_X(X)} \right].$$

The random variable inside the expectation is the log-likelihood ratio favoring $P$ over $Q$ after seeing the outcome. Thus $D(P\Vert Q)$ is the average log evidence per sample, assuming the data really come from $P$.

For independent samples,

$$X_1,\ldots,X_n\sim P,$$

the product distributions satisfy

$$D(P^{\otimes n}\Vert Q^{\otimes n}) = nD(P\Vert Q).$$

This additivity is why relative entropy controls exponential rates in hypothesis testing and large-deviation theory.

Positivity

The fundamental inequality is

$$D(P\Vert Q)\ge0.$$

This is often called Gibbs’ inequality. Equality holds exactly when $P=Q$.

A direct proof uses

$$-\log u\ge1-u, \qquad u>0.$$

Assume first that $P\ll Q$. Then

$$\begin{aligned} D(P\Vert Q) &= \sum_{p_x>0} p_x \left[ - \log \frac{q_x}{p_x} \right]\\ &\ge \sum_{p_x>0} p_x \left[ 1-\frac{q_x}{p_x} \right]\\ &= 1- \sum_{p_x>0}q_x\\ &\ge 0. \end{aligned}$$

If $P$ and $Q$ have the same support, the last line is exactly zero only when $q_x/p_x=1$ for every $x$ with $p_x>0$. If $Q$ has extra probability outside the support of $P$, the inequality is strict. Thus equality holds exactly for identical distributions.

Not a Distance

Despite the word “divergence,” relative entropy is not a distance metric. In general,

$$D(P\Vert Q)\ne D(Q\Vert P).$$

It also does not satisfy the triangle inequality. The order of the arguments matters:

• $D(P\Vert Q)$ averages log evidence using data drawn from $P$;
• $D(Q\Vert P)$ averages log evidence using data drawn from $Q$.

These can be very different, especially when one distribution assigns small or zero probability to events that are plausible under the other.

Relation to Entropy and Cross-Entropy

For discrete distributions, define the cross-entropy

$$H(P,Q) = - \sum_x p_x\log q_x.$$

The Shannon entropy of $P$ is

$$H(P) = - \sum_x p_x\log p_x.$$

Then

$$D(P\Vert Q) = H(P,Q)-H(P).$$

Thus relative entropy is the extra average code length incurred when using a code optimized for $Q$ while the true distribution is $P$, in the idealized Shannon coding interpretation.

Bernoulli Example

Let $P$ and $Q$ be two Bernoulli distributions:

$$P(1)=q, \qquad Q(1)=r.$$

Then

$$D(P\Vert Q) = q\log\frac{q}{r} + (1-q)\log\frac{1-q}{1-r}.$$

This expression is not symmetric under swapping $q$ and $r$.

If $q=1$ and $r\lt1$, then

$$D(P\Vert Q)=\log\frac{1}{r}.$$

If $q>0$ and $r=0$, then $D(P\Vert Q)=+\infty$, because $Q$ declares an event impossible that $P$ can produce.

Continuous Distributions

For probability densities $p(x)$ and $q(x)$ with respect to the same reference measure, the relative entropy is

$$D(P\Vert Q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx,$$

again with the condition that $P$ be absolutely continuous with respect to $Q$.

Unlike differential entropy, relative entropy is invariant under smooth one-to-one changes of variables. If both densities transform with the same Jacobian, the Jacobian factors cancel inside the ratio

$$\frac{p(x)}{q(x)}.$$

This is one reason relative entropy is often more robust than differential entropy in continuum calculations.

For two one-dimensional Gaussian distributions with the same variance,

$$P=\mathcal N(\mu_0,\sigma^2), \qquad Q=\mathcal N(\mu_1,\sigma^2),$$

one finds, using natural logarithms,

$$D(P\Vert Q) = \frac{(\mu_0-\mu_1)^2}{2\sigma^2}.$$

For unequal variances the expression has additional scale terms; see the Gaussian background in Gaussian Distributions. The local curvature of this comparison is Fisher Information.

Mutual Information as Relative Entropy

Let $X$ and $Y$ have joint distribution $P_{XY}$ and marginals $P_X$, $P_Y$. The classical mutual information is

$$I(X:Y) = D \bigl( P_{XY} \Vert P_XP_Y \bigr).$$

Written out,

$$I(X:Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p_X(x)p_Y(y)}.$$

This says that mutual information measures how distinguishable the actual joint distribution is from the uncorrelated product distribution with the same marginals.

The nonnegativity of relative entropy immediately gives

$$I(X:Y)\ge0,$$

with equality exactly when $X$ and $Y$ are independent.

Bayesian and Statistical Meaning

Relative entropy is not the same as Bayesian updating, but it appears naturally in likelihood comparisons.

Suppose two models assign probabilities $p(D)$ and $q(D)$ to data $D$. The log-likelihood ratio is

$$\log\frac{p(D)}{q(D)}.$$

If repeated data are generated from $P$, the average log-likelihood ratio per sample approaches

$$D(P\Vert Q)$$

under standard law-of-large-numbers assumptions. Thus KL divergence measures the asymptotic expected evidence rate against model $Q$ when $P$ is the data-generating distribution.

In parameter estimation, minimizing $D(P_{\mathrm{true}}\Vert Q_\theta)$ is the idealized population version of maximum-likelihood fitting when the model family $Q_\theta$ may be misspecified.

Quantum Relative Entropy Preview

For finite-dimensional density operators $\rho$ and $\sigma$, the quantum relative entropy is

$$D(\rho\Vert\sigma) = \operatorname{Tr} \bigl[ \rho(\log\rho-\log\sigma) \bigr],$$

provided the support of $\rho$ is contained in the support of $\sigma$. Otherwise it is infinite.

If $\rho$ and $\sigma$ commute, they can be diagonalized in the same basis, and the formula reduces to the classical KL divergence between their eigenvalue distributions.

Quantum mutual information can be written as

$$I(A:B)_\rho = D \bigl( \rho_{AB} \Vert \rho_A\otimes\rho_B \bigr).$$

That identity is developed in Mutual Information. The basic von Neumann entropy entering the formula is introduced in Entropy Overview.

In continuum and field-theoretic settings, relative entropy is often better behaved than individual entanglement entropies, but the correct formulation requires algebras, regulators, and domain care. The finite-QM bridge is Entanglement in QFT Preview.

Common Mistakes

• Treating $D(P\Vert Q)$ as a symmetric distance.
• Forgetting the support condition; assigning zero probability to a possible event can make the divergence infinite.
• Swapping the order of $P$ and $Q$ without changing the interpretation.
• Calling KL divergence an entropy of a single distribution rather than a comparison between two distributions.
• Confusing differential entropy, which is coordinate dependent, with relative entropy, whose density ratio cancels Jacobians.
• Using the quantum formula without checking noncommutativity and support.
• Saying relative entropy is “small” without specifying the logarithm base and scale of the comparison.

Cross-Links

• Entropy
• Conditional Probability
• Bayes’ Rule
• Gaussian Distributions
• Fisher Information
• Mutual Information
• Entropy Overview
• Entanglement in QFT Preview
• Classical Information Review applies Kullback–Leibler divergence and its support convention to declared classical comparator audits; this page retains the mathematical and statistical interpretation.

References

• S. Kullback and R. A. Leibler, “On Information and Sufficiency,” Annals of Mathematical Statistics 22, 79–86, 1951.
• T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
• I. Csiszar and P. C. Shields, “Information Theory and Statistics: A Tutorial,” Foundations and Trends in Communications and Information Theory 1, 417–528, 2004.
• D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.
• M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information, Cambridge University Press, 2010.
• J. Watrous, The Theory of Quantum Information, Cambridge University Press, 2018.

Exercises

1. Compute $D(P\Vert Q)$ for Bernoulli distributions with $P(1)=q$ and $Q(1)=r$.

Solution

The two outcomes have probabilities

$$P(1)=q, \qquad P(0)=1-q,$$

and

$$Q(1)=r, \qquad Q(0)=1-r.$$

Therefore

$$D(P\Vert Q) = q\log\frac{q}{r} + (1-q)\log\frac{1-q}{1-r},$$

with the usual zero and support conventions.

2. Show that $D(P\Vert Q)=0$ if $P=Q$.

Solution

If $P=Q$, then $p_x=q_x$ for every $x$. Hence

$$\log\frac{p_x}{q_x} = \log1 = 0$$

for every term with $p_x>0$. Terms with $p_x=0$ also contribute $0$. Therefore

$$D(P\Vert Q)=0.$$

The converse follows from Gibbs’ inequality: equality in the positivity proof requires the distributions to agree.

3. Prove additivity for independent product distributions.

Solution

Let $P_1,Q_1$ be distributions on $X$ and $P_2,Q_2$ distributions on $Y$. For the product distributions,

$$p(x,y)=p_1(x)p_2(y), \qquad q(x,y)=q_1(x)q_2(y).$$

Then

$$\begin{aligned} D(P_1P_2\Vert Q_1Q_2) &= \sum_{x,y} p_1(x)p_2(y) \log \frac{p_1(x)p_2(y)}{q_1(x)q_2(y)}\\ &= \sum_{x,y} p_1(x)p_2(y) \left[ \log\frac{p_1(x)}{q_1(x)} + \log\frac{p_2(y)}{q_2(y)} \right]\\ &= D(P_1\Vert Q_1)+D(P_2\Vert Q_2). \end{aligned}$$

4. A fair bit $X$ is copied to $Y=X$. Show that $I(X:Y)=1$ bit using relative entropy.

Solution

The joint distribution has

$$p(0,0)=p(1,1)=\frac12,$$

and the other two joint outcomes have probability $0$. The marginals are uniform, so the product distribution assigns probability $1/4$ to each pair.

Therefore, in base $2$,

$$\begin{aligned} I(X:Y) &= \sum_{x,y} p(x,y) \log_2 \frac{p(x,y)}{p_X(x)p_Y(y)}\\ &= \frac12\log_2\frac{1/2}{1/4} + \frac12\log_2\frac{1/2}{1/4}\\ &= 1. \end{aligned}$$

The copied bit contains one bit of shared information.

5. Derive $D(P\Vert Q)$ for two Gaussians with the same variance.

Solution

Let

$$P=\mathcal N(\mu_0,\sigma^2), \qquad Q=\mathcal N(\mu_1,\sigma^2).$$

The log density ratio is

$$\log\frac{p(x)}{q(x)} = - \frac{(x-\mu_0)^2}{2\sigma^2} + \frac{(x-\mu_1)^2}{2\sigma^2}.$$

Taking expectation under $P$ gives

$$D(P\Vert Q) = \frac{1}{2\sigma^2} \mathbb E_P \left[ (X-\mu_1)^2-(X-\mu_0)^2 \right].$$

Since

$$\mathbb E_P[(X-\mu_0)^2]=\sigma^2,$$

and

$$\mathbb E_P[(X-\mu_1)^2] = \sigma^2+(\mu_0-\mu_1)^2,$$

one obtains

$$D(P\Vert Q) = \frac{(\mu_0-\mu_1)^2}{2\sigma^2}.$$