Entropy

Entropy is a function of a probability distribution that measures average uncertainty, average information gained on learning an outcome, or average surprise, depending on context.

For a discrete random variable $X$ with probabilities $p_x=\mathbb P(X=x)$, the Shannon entropy is

$$H(X) = - \sum_x p_x\log p_x.$$

The entropy depends on the probabilities, not on the names of the outcomes. A deterministic variable has zero entropy. A uniform distribution over $n$ possible outcomes has entropy $\log n$.

This page gives the classical probability version. The density-operator version is Entropy Overview, and reduced-state entropy in composite quantum systems is Subsystem Entropy.

Discrete Shannon Entropy

Let $X$ take values in a finite or countable set with probabilities

$$p_x\ge0, \qquad \sum_x p_x=1.$$

The Shannon entropy is

$$H(X) = - \sum_x p_x\log p_x,$$

with the convention

$$0\log0=0.$$

This convention is the limit

$$\lim_{p\to0^+}p\log p=0.$$

The logarithm base fixes the units:

• $\log_2$ gives entropy in bits;
• $\ln$ gives entropy in nats;
• $\log_b$ gives entropy in units of $\log b$.

Changing the base rescales entropy by a constant. For example,

$$H_{\mathrm{bits}}(X) = \frac{H_{\mathrm{nats}}(X)}{\ln2}.$$

Entropy as Expected Surprise

The information content, or surprise, of an outcome with probability $p$ is

$$-\log p.$$

Rare outcomes carry larger surprise than common outcomes. The entropy is the average surprise:

$$H(X) = \mathbb E[-\log p_X(X)].$$

Written out,

$$\mathbb E[-\log p_X(X)] = \sum_x p_x[-\log p_x] = - \sum_x p_x\log p_x.$$

This interpretation is useful but should not be overextended. Entropy is not itself a random measurement outcome. It is a functional of the whole distribution.

Deterministic and Uniform Cases

If $X$ is deterministic, then one outcome has probability $1$ and all others have probability $0$. Therefore

$$H(X)=0.$$

If $X$ is uniform on $n$ outcomes, then $p_x=1/n$ and

$$\begin{aligned} H(X) &= - \sum_{x=1}^n \frac1n\log\frac1n\\ &= \log n. \end{aligned}$$

For a fixed finite set of $n$ outcomes, this is the maximum possible entropy. Intuitively, uncertainty is largest when no outcome is favored.

Binary Entropy

For a two-outcome random variable with probabilities $q$ and $1-q$, the entropy in bits is

$$H_2(q) = - q\log_2 q - (1-q)\log_2(1-q).$$

It satisfies

$$H_2(0)=H_2(1)=0, \qquad H_2(1/2)=1.$$

The binary entropy is symmetric:

$$H_2(q)=H_2(1-q).$$

It is the classical template behind the entropy of a diagonal qubit density matrix with eigenvalues $q$ and $1-q$.

Joint Entropy

For a pair of discrete random variables $(X,Y)$ with joint probabilities

$$p_{xy} = \mathbb P(X=x,Y=y),$$

the joint entropy is

$$H(X,Y) = - \sum_{x,y}p_{xy}\log p_{xy}.$$

If $X$ and $Y$ are independent, then

$$p_{xy}=p_xp_y,$$

and

$$H(X,Y)=H(X)+H(Y).$$

If $Y=X$ exactly, then the pair $(X,Y)$ contains no more uncertainty than $X$ alone:

$$H(X,Y)=H(X).$$

Thus joint entropy counts uncertainty in the joint outcome, not the number of variables written down.

Conditional Entropy

The conditional entropy of $Y$ given $X$ is

$$H(Y\mid X) = \sum_x p_x H(Y\mid X=x).$$

Equivalently,

$$H(Y\mid X) = - \sum_{x,y} p_{xy}\log p_{y\mid x},$$

where

$$p_{y\mid x} = \mathbb P(Y=y\mid X=x)$$

when $p_x>0$.

The chain rule is

$$H(X,Y) = H(X)+H(Y\mid X).$$

For ordinary classical variables,

$$H(Y\mid X)\ge0.$$

If $Y$ is determined by $X$, then $H(Y\mid X)=0$. If $X$ and $Y$ are independent, then $H(Y\mid X)=H(Y)$.

Quantum conditional entropy can be negative; that is a property of von Neumann entropy for composite quantum states, not of classical conditional Shannon entropy.

Mutual Information Preview

Classical mutual information is

$$I(X:Y) = H(X)+H(Y)-H(X,Y).$$

Using the chain rule,

$$I(X:Y) = H(Y)-H(Y\mid X).$$

It measures the reduction in uncertainty about one variable obtained by learning the other. It vanishes when $X$ and $Y$ are independent.

Equivalently, mutual information is a relative entropy comparing the joint distribution with the product of its marginals.

The quantum analogue uses von Neumann entropies of density operators:

$$I(A:B)_\rho = S(\rho_A)+S(\rho_B)-S(\rho_{AB}).$$

For the quantum version and its interpretation as total correlation, see Mutual Information.

Entropy and Coarse Graining

Entropy depends on the variables and distinctions being used. If a fine-grained random variable $X$ is mapped to a coarser variable

$$Y=g(X),$$

then $Y$ cannot contain more Shannon entropy than $X$:

$$H(Y)\le H(X).$$

Coarse graining can merge outcomes and discard distinctions. The lost information is zero only when $g$ is one-to-one on the support of the distribution.

In physics, this warning matters because the entropy assigned to a description depends on what macroscopic variables, measurement outcomes, subsystem splits, or preparation records are being retained.

Differential Entropy Caution

For a continuous random variable with density $f_X(x)$, one can define the differential entropy

$$h(X) = - \int f_X(x)\log f_X(x)\,dx,$$

when the integral is meaningful.

This resembles Shannon entropy, but it is not the same kind of invariant uncertainty measure. There are three major cautions.

First, a density has units. Taking the logarithm of a dimensionful density is shorthand for a convention involving a reference measure or unit scale.

Second, differential entropy changes under coordinate scaling. If

$$Y=aX, \qquad a\ne0,$$

then

$$h(Y)=h(X)+\log\lvert a\rvert.$$

Changing from meters to centimeters changes the numerical differential entropy.

Third, differential entropy can be negative. A sharply localized density can have $h(X)\lt0$ in chosen units. This is not a paradox; it means differential entropy is not a direct continuous analogue of the nonnegative discrete entropy.

For a Gaussian density with variance $\sigma^2$, the differential entropy is

$$h(X) = \frac12 \log \left( 2\pi e\,\sigma^2 \right),$$

with the same unit-scale caution. The probability-density facts behind this formula are collected in Gaussian Distributions.

Quantum-Mechanics Interpretation

There are several different entropy-like quantities in quantum mechanics. Keeping them separate prevents many mistakes.

The Shannon entropy of a measurement is the entropy of the Born probabilities for that chosen measurement:

$$H(A) = - \sum_a p(a)\log p(a), \qquad p(a)=\langle\psi\vert P_a\lvert\psi\rangle$$

for a projective measurement on a pure state.

The von Neumann entropy of a density operator is

$$S(\rho) = - \operatorname{Tr}(\rho\log\rho).$$

If $\rho$ has eigenvalues $\lambda_k$, then

$$S(\rho) = - \sum_k\lambda_k\log\lambda_k.$$

Thus von Neumann entropy is the Shannon entropy of the eigenvalue distribution of $\rho$, not the Shannon entropy of an arbitrary measurement outcome distribution.

A pure state can have zero von Neumann entropy but nonzero measurement entropy in a basis where the outcome is not certain. For example,

$$\lvert+\rangle = \frac{\lvert0\rangle+\lvert1\rangle}{\sqrt2}$$

has zero von Neumann entropy as a pure state, but a computational-basis measurement has one bit of Shannon entropy.

For reduced states of composite systems, von Neumann entropy can quantify local mixedness, pure-state bipartite entanglement, thermal uncertainty, or correlation-derived quantities depending on the context. The context must be stated.

Common Mistakes

• Forgetting to state the logarithm base.
• Treating $0\log0$ as undefined instead of using its limiting value $0$.
• Calling entropy a property of a single outcome rather than of a distribution.
• Confusing Shannon entropy of a chosen measurement with von Neumann entropy of a quantum state.
• Assuming a positive subsystem entropy always means entanglement, even for mixed joint states.
• Treating differential entropy as coordinate invariant.
• Forgetting that differential entropy can be negative.
• Interpreting entropy without specifying the retained variables, measurement, subsystem split, or coarse graining.

Cross-Links

• Probability Spaces, Light Version
• Random Variables
• Probability Densities
• Expectation Values
• Conditional Probability
• Bayes’ Rule
• Relative Entropy
• Gaussian Distributions
• Entropy Overview
• Subsystem Entropy
• Mutual Information
• Classical Information Review applies Shannon and binary entropy to source-coding and fair-comparator checks; this page retains their definitions and derivations.

References

• C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal 27, 379–423 and 623–656, 1948.
• A. I. Khinchin, Mathematical Foundations of Information Theory, Dover, 1957.
• T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
• D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.
• J. von Neumann, Mathematical Foundations of Quantum Mechanics, Princeton University Press, 1955.
• M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information, Cambridge University Press, 2010.

Exercises

1. Compute the entropy in bits of a biased coin with $\mathbb P(H)=q$.

Solution

The distribution has probabilities $q$ and $1-q$, so

$$H = - q\log_2 q - (1-q)\log_2(1-q).$$

This is the binary entropy $H_2(q)$.

2. Show that the uniform distribution on $n$ outcomes has entropy $\log n$.

Solution

For a uniform distribution, $p_x=1/n$ for each of the $n$ outcomes. Therefore

$$\begin{aligned} H &= - \sum_{x=1}^n \frac1n\log\frac1n\\ &= - \log\frac1n\\ &= \log n. \end{aligned}$$

3. Let $X$ be a fair bit and let $Y=X$. Compute $H(X)$, $H(Y)$, and $H(X,Y)$ in bits.

Solution

Since $X$ is a fair bit,

$$H(X)=1.$$

The same is true for $Y$:

$$H(Y)=1.$$

But the joint pair has only two possible outcomes, $(0,0)$ and $(1,1)$, each with probability $1/2$. Hence

$$H(X,Y)=1.$$

The two variables are perfectly correlated, so the joint uncertainty is not two bits.

4. If $Y=aX$ with $a\ne0$, show that $h(Y)=h(X)+\log\lvert a\rvert$.

Solution

The density of $Y$ is

$$f_Y(y) = \frac{1}{\lvert a\rvert} f_X \left( \frac{y}{a} \right).$$

Then

$$\begin{aligned} h(Y) &= - \int f_Y(y)\log f_Y(y)\,dy\\ &= - \int f_X(x) \log \left[ \frac{1}{\lvert a\rvert}f_X(x) \right]dx\\ &= - \int f_X(x)\log f_X(x)\,dx + \log\lvert a\rvert \int f_X(x)\,dx\\ &= h(X)+\log\lvert a\rvert. \end{aligned}$$

5. A qubit is in the pure state $\lvert+\rangle=(\lvert0\rangle+\lvert1\rangle)/\sqrt2$. Compare its von Neumann entropy with the Shannon entropy of a computational-basis measurement.

Solution

The density operator $\rho=\lvert+\rangle\langle+\rvert$ is pure, so its eigenvalues are $1$ and $0$. Therefore

$$S(\rho)=0.$$

A computational-basis measurement has probabilities

$$p(0)=p(1)=\frac12.$$

The Shannon entropy of that measurement is

$$H = - \frac12\log_2\frac12 - \frac12\log_2\frac12 = 1$$

bit. The state entropy and the measurement-outcome entropy answer different questions.