1.14 Cross-entropy and KL divergence: study notes

Learn AI by Story

Concept 1.14, comparing distributions. Video story key: p114. Previous: 1.13 Entropy. Next: 1.15 Eigenvectors and SVD.

1. The idea in one sentence

Cross-entropy is the average surprise (cost in bits) you pay when the world follows distribution P but you predict with distribution Q, and the KL divergence is the extra cost caused purely by Q being wrong, so training a model to lower cross-entropy is the same as pulling its beliefs toward the truth.

2. The story from the video

Omar believes a coin lands heads 90 percent of the time, but in truth it is 50-50. Calling the truth P and the belief Q, the video adds up the surprise Omar suffers on real coin flips: 0.5 × 0.152 for heads plus 0.5 × 3.322 for tails, which is 1.737 bits. The truth alone costs only 1 bit (its entropy), so the wrong belief wastes 0.737 bits. That waste is the KL divergence, and the identity cross-entropy = entropy + KL ties the three numbers together. The video then shows the classifier loss curve (0.9 on the right answer costs 0.15 bits, 0.1 costs 3.32 bits) and the AI uses: classifiers, language models, and keeping a model close to a reference.

3. Key definitions

Probability distribution. A list of probabilities for every possible outcome, adding to 1 (see 1.10 distributions). Plain English: the rules of how likely each thing is.
Self-information (surprise). For an outcome with probability p, I = −log2 p bits. Plain English: rare events are more surprising; a certain event carries zero surprise.
Bit and nat. A bit is surprise measured with log base 2; a nat uses the natural logarithm (base e). Conversion: bits = nats / ln 2, and one nat is about 1.443 bits. Plain English: same quantity, different ruler.
Entropy H(P). H(P) = −Σ p log p, the expected surprise when events follow P and you also predict with P (see 1.13). Plain English: the unavoidable average surprise of the truth.
Cross-entropy H(P, Q). H(P, Q) = −Σ p(x) log q(x). Events are drawn from P but the surprise is charged using Q. Plain English: the cost of using the wrong belief. It also equals the average number of bits needed to encode data from P with a code designed for Q.
KL divergence D_KL(P‖Q). D_KL(P‖Q) = Σ p(x) log( p(x) / q(x) ) = H(P, Q) − H(P). Plain English: the extra cost of believing Q when the truth is P.
Truth P and model Q. By convention P is the true (or data) distribution and Q is the model's belief. The order matters, as explained below.
One-hot label. A label vector with a 1 on the correct class and 0 elsewhere. It is a distribution that puts all probability on one answer. Plain English: "the answer is exactly this one."
Loss function. A number the training procedure tries to make small. Plain English: the score for how wrong the model is.
Log-likelihood. The log of the probability the model gave to the observed data. Maximising it is the same as minimising the negative log-likelihood. Plain English: reward the model for giving high probability to what really happened.
Divergence versus distance. A distance (metric) is symmetric and obeys the triangle inequality. KL is neither, so it is called a divergence. Plain English: a one-way measure of mismatch.
Softmax. A function that turns a list of raw scores into probabilities that sum to 1. Plain English: it lets a network output a distribution Q for the loss to compare with P.

4. Detailed step-by-step explanation

Step 1: Surprise from entropy

In 1.13 we learned that an outcome with probability p has surprise −log2 p bits: 1 bit for p = 1/2, 3 bits for p = 1/8, 6 bits for p = 1/64. Entropy is the average of that surprise under the true probabilities. For a fair coin the average is 1 bit; for a coin that lands heads 90 percent of the time it is only 0.469 bits, because the outcome is mostly predictable.

Step 2: Two distributions, not one

Entropy uses a single distribution twice: once to decide how often each event happens, once to decide how surprising each event is. Now separate the two roles. Reality (P) decides how often each event happens. Your model (Q) decides how surprised you are. The average surprise you actually experience is the cross-entropy. The word "cross" means the two distributions are crossed over: P supplies the weights, Q supplies the logarithms.

Step 3: The coin from the video

P = (heads 0.5, tails 0.5). Q = (heads 0.9, tails 0.1). Surprise under Q for heads is −log2 0.9 = 0.152 bits; for tails it is −log2 0.1 = 3.322 bits. Weight each by how often it really happens: 0.5 × 0.152 = 0.076 and 0.5 × 3.322 = 1.661. Add them: cross-entropy = 1.737 bits. The tails term dominates: Omar thinks tails is rare, so each real tails shocks him, and tails happens half the time.

Step 4: The floor and the gap

If Omar had believed the truth, his average surprise would have been the entropy of P, which is 1 bit. This is a floor: no belief can give a lower average than the truth does. So the cross-entropy 1.737 splits into an unavoidable part (1.000) and an avoidable part (0.737). The avoidable part is the KL divergence: D_KL(P‖Q) = 1.737 − 1.000 = 0.737 bits. This gives the identity H(P, Q) = H(P) + D_KL(P‖Q).

H(P,Q) = H(P) + D_{KL}(P \Vert Q), \qquad D_{KL}(P \Vert Q) = \sum_x p(x)\log\frac{p(x)}{q(x)}

Step 5: Why KL is never negative

KL measures an average of log(p/q) weighted by p. A standard result (often called Gibbs' inequality) says this average is never below zero, and it equals zero only when Q is exactly P. Intuitively: you cannot beat a source's own entropy by using a wrong model of it. The d2l.ai appendix states these properties: non-negative, zero only when P = Q, and non-symmetric.

Step 6: Why the direction matters

The video reports D_KL(P‖Q) = 0.737 but D_KL(Q‖P) = 0.531. In the first, the weights come from the truth; in the second they come from the belief. They answer different questions: "how costly is it to use Q when reality is P?" versus "how costly is it to use P when reality is Q?" Because the numbers differ, KL is not a distance. This is why one writes D_KL(P‖Q) with the double bar and keeps track of which side is which.

Step 7: Why training uses cross-entropy, not KL

To train a model we adjust Q. The entropy H(P) of the data does not depend on the model parameters, so it is a constant. Since H(P, Q) = H(P) + D_KL(P‖Q), minimising cross-entropy over the model is exactly equivalent to minimising KL. Cross-entropy is preferred because it can be computed from samples alone: we never need to know H(P), only to average −log q over data we have. The d2l.ai appendix lists three equivalent goals: maximising the expected log q under P, minimising cross-entropy, and minimising KL.

Step 8: The classification special case

In classification the truth for one example is a one-hot label: P puts probability 1 on the correct class. The cross-entropy then collapses to a single term, −log q(correct class). So the loss for one example is simply the surprise of the model at the right answer. With q = 0.9 the loss is 0.152 bits (0.105 nats); q = 0.5 gives 1 bit (0.693 nats); q = 0.1 gives 3.322 bits (2.303 nats); q = 0.99 gives 0.0145 bits. The loss climbs steeply as the right answer's probability approaches zero: confident and wrong costs the most. For two classes the formula is the binary cross-entropy −[y log π + (1 − y) log(1 − π)], where π is the probability given to the positive class and y is 0 or 1.

Step 9: The link to maximum likelihood

Summing −log q(correct class) over a dataset is the negative log-likelihood. So training with cross-entropy is maximum likelihood estimation in disguise: choose the model that makes the observed labels most probable. The Greenplace article notes the same equivalence.

Step 10: Nats in practice

Most libraries use the natural logarithm, so reported losses are in nats. Divide by ln 2 (about 0.693) to get bits. A loss of 0.693 nats is exactly 1 bit, the loss of a coin-flip guess between two classes.

5. Everything the video skipped

Infinite KL. If the truth gives positive probability to an outcome that the model gives probability 0, the term p log(p/0) is infinite, so KL is infinite. Practical lesson: never let a model assign exactly zero to something that can happen. This is one reason softmax outputs are strictly positive and why implementations work with log-probabilities.

Forward and reverse KL. D_KL(P‖Q) (forward, truth first) punishes Q for missing anywhere P has mass, so a fitted Q tends to spread out and cover all of P's modes ("mode-covering"). D_KL(Q‖P) (reverse) punishes Q for putting mass where P has little, so a fitted Q tends to lock onto one mode ("mode-seeking"). This is a common description in the literature; the exact behaviour depends on the model family.

Mutual information. The d2l.ai appendix notes that mutual information I(X, Y) equals the KL divergence between the joint distribution and the product of the marginals. It measures how far X and Y are from being independent (see 1.08 probability basics).

Continuous distributions. For densities, sums become integrals and the same formulas hold; KL between two Gaussians has a closed form, which is why variational autoencoders (VAEs) use it.

Symmetric variants. Because KL is one-way, people sometimes average the two directions or use other measures (such as Jensen-Shannon divergence, a symmetrised and smoothed relative of KL). These are pointers only.

Links to the chain rule and Bayes. Probabilities of independent events multiply, so their logs add (see 1.07 chain rule and 1.09 Bayes). That is why a dataset's loss is the sum or average of per-example losses.

Limitations and trade-offs.

Cross-entropy only looks at the probability given to the correct class, not at which wrong classes were favoured.
It rewards confident models, so overconfident predictions on noisy labels can be penalised heavily, which motivates tricks like label smoothing (replacing the one-hot target with a slightly softened distribution).
KL is not a metric, so "smaller KL" does not behave like "closer" in every geometric sense.
A low cross-entropy does not by itself mean high accuracy; the two usually move together but are different quantities.

Roadmap links. Earlier: 1.07 chain rule, 1.08 probability basics, 1.09 Bayes, 1.10 distributions, 1.11 mean, variance and correlation, 1.13 entropy. Later: 1.15 eigenvectors and SVD, and 1.17 vector norms, which also give ways to measure size and difference, but for vectors rather than distributions.

6. Worked examples

All arithmetic below was checked in Python.

Worked example A: the video's coin

P = (0.5, 0.5), Q = (0.9, 0.1).
H(P) = −(0.5 log2 0.5 + 0.5 log2 0.5) = 1 bit.
Surprises under Q: −log2 0.9 = 0.152, −log2 0.1 = 3.322.
Cross-entropy: 0.5 × 0.152 + 0.5 × 3.322 = 0.076 + 1.661 = 1.737 bits.
KL(P‖Q) = 1.737 − 1 = 0.737 bits.
Reverse: KL(Q‖P) = 0.9 log2(0.9/0.5) + 0.1 log2(0.1/0.5) = 0.9 × 0.848 + 0.1 × (−2.322) = 0.763 − 0.232 = 0.531 bits.
In nats, multiply by ln 2: cross-entropy 1.204 nats, KL 0.511 nats.

Worked example B: three-class weather (illustrative numbers)

Suppose the real chance of weather is P = (sunny 0.7, cloudy 0.2, rainy 0.1), and a forecaster believes Q = (0.5, 0.3, 0.2).

H(P) = −(0.7 log2 0.7 + 0.2 log2 0.2 + 0.1 log2 0.1) = 0.360 + 0.464 + 0.332 = 1.157 bits.
Cross-entropy: 0.7 × 1 + 0.2 × 1.737 + 0.1 × 2.322 = 0.700 + 0.347 + 0.232 = 1.280 bits.
KL(P‖Q) = 1.280 − 1.157 = 0.123 bits.
Reverse KL(Q‖P) = 0.133 bits. Different from the forward value, though close here because the beliefs are not far apart.
Reading: the forecaster wastes about 0.12 bits per day compared with knowing the truth.

Worked example C: a classifier loss on a small batch (illustrative probabilities)

A classifier sees three pictures. The probabilities it gives the correct class are 0.8, 0.6 and 0.1.

Losses in nats: −ln 0.8 = 0.223; −ln 0.6 = 0.511; −ln 0.1 = 2.303.
Batch average: (0.223 + 0.511 + 2.303) / 3 = 1.012 nats.
The single bad prediction (0.1) contributes 2.303 out of 3.037, about 76 percent of the total. That is why training pushes hardest on its worst mistakes.
In bits the average is 1.012 / 0.693 = 1.46 bits.

Worked example D: the cost of near-certainty (illustrative beliefs)

Keep P = (0.5, 0.5). If Q = (0.99, 0.01): cross-entropy = 0.5 × 0.0145 + 0.5 × 6.644 = 3.329 bits, so KL = 2.329 bits, more than three times Omar's 0.737. If Q = (1.0, 0.0), then tails has q = 0 yet p = 0.5, so KL is infinite. Being absolutely sure and wrong is unboundedly costly.

Worked example E: the textbook five-outcome case

From the Greenplace article: P = (0.1, 0.2, 0.4, 0.2, 0.1) and Q uniform over five outcomes. Cross-entropy = log2 5 = 2.32 bits; H(P) = 2.12 bits; KL = 0.20 bits. A Q closer to P, (0.15, 0.175, 0.35, 0.175, 0.15), lowers the cross-entropy to 2.16 bits, as the article reports.

7. Comparison tables and diagrams

Quantity	Formula	Needs	Plain English	Symmetric?
Entropy H(P)	−Σ p log p	P only	Unavoidable average surprise	not applicable
Cross-entropy H(P, Q)	−Σ p log q	P and Q	Average surprise using the wrong belief	No
KL D_KL(P‖Q)	Σ p log(p/q)	P and Q	Extra cost of the wrong belief	No
Reverse KL D_KL(Q‖P)	Σ q log(q/p)	P and Q	The cost the other way round	No

Probability given to right answer	Loss in bits	Loss in nats
0.99	0.0145	0.010
0.9	0.152	0.105
0.5	1.000	0.693
0.1	3.322	2.303
0.05	4.322	2.996

Simple diagram of the identity (bar widths illustrative, using the coin numbers):

Cross-entropy 1.737 bits
[#########################][###########]
   entropy of truth = 1.000    KL gap = 0.737

Question	Entropy	Cross-entropy	KL
Can it be negative (discrete case)?	No	No	No
Zero when?	One outcome is certain	Only if P is certain and Q matches	P = Q
Used as a loss?	Rarely	Yes, very common	Yes, in distillation and VAEs

8. Common mistakes and myths

Myth: KL is a distance. It is not symmetric and does not obey the triangle inequality. The video's numbers 0.737 and 0.531 show the asymmetry.
Mistake: swapping P and Q. The truth (or data) goes first. Swapping changes both the value and the behaviour of the fitted model.
Mistake: mixing bits and nats. A reported loss of 0.693 is one bit worth of ignorance if it is in nats. Check which logarithm your library uses.
Myth: cross-entropy of 0 is always reachable. Cross-entropy cannot go below H(P). For data with ambiguous inputs the floor is above zero; only for a one-hot truth is the floor 0.
Mistake: giving a model exact zeros. Probability 0 on a possible event makes KL infinite and log(0) undefined. Use softmax and numerically stable log-softmax.
Myth: lower loss always means better accuracy. A model can improve its loss by becoming more confident on right answers without changing a single decision.
Mistake: forgetting that an average hides outliers. One 0.1-probability miss can dominate a batch, as example C shows.

9. Real-life uses with concrete examples

Image and text classifiers. The standard loss is cross-entropy between the one-hot label and the softmax output. For a photo of a cat, the loss is −log of the probability the model gives to "cat".
Language models. Training minimises cross-entropy on each next word (token) of the text. The related quantity perplexity is 2 raised to the cross-entropy in bits (e raised to it in nats), roughly how many equally likely words the model is hesitating between.
Knowledge distillation. A small student model is trained to match the output distribution of a large teacher by minimising KL between them.
Variational autoencoders. The VAE loss includes a KL term that keeps the learned latent distribution close to a simple prior.
Fine-tuning with a reference. Some fine-tuning methods add a KL penalty so the updated model stays close to a reference model (stated generically).
Compression. Cross-entropy is the average code length when you build a code for Q but the data come from P, so a wrong model literally costs extra bits.
Model comparison. Lower held-out cross-entropy indicates a model that assigns more probability to real data.

10. Self-check questions

What does cross-entropy H(P, Q) measure? The average surprise (in bits or nats) when events follow P but you charge surprise using Q.
State the identity relating cross-entropy, entropy and KL. H(P, Q) = H(P) + D_KL(P‖Q).
In the video, why is the entropy of P equal to 1 bit? P is a fair coin: 0.5 × 1 + 0.5 × 1 = 1 bit.
Compute the cross-entropy for P = (0.5, 0.5), Q = (0.9, 0.1). 0.5 × 0.152 + 0.5 × 3.322 = 1.737 bits.
Is KL symmetric? Give evidence. No. The video's coin gives 0.737 one way and 0.531 the other.
When is KL exactly zero? Only when P and Q are identical.
Why can we minimise cross-entropy instead of KL during training? H(P) does not depend on the model, so the two differ by a constant and have the same minimiser.
What is the loss for one example when the model gives the right class probability 0.5? 1 bit, or 0.693 nats.
What happens to KL if Q gives probability zero to something P says can happen? KL becomes infinite.
Convert a loss of 2.303 nats to bits. 2.303 / 0.693 = 3.32 bits.

11. Key takeaways, go deeper, and what is next

Key takeaways

Cross-entropy is the cost of using the wrong belief; KL is the extra cost beyond the truth's own entropy.
H(P, Q) = H(P) + D_KL(P‖Q), so reducing one reduces the other for a fixed truth.
KL is not symmetric, is zero only when P = Q, and is infinite when Q misses something possible.
Classification loss is just −log of the probability given to the right answer: confident and wrong costs the most.
Libraries use nats; divide by ln 2 for bits.

Go deeper

Work out forward versus reverse KL when fitting a single bell curve to a two-bump distribution.
Compute perplexity from a cross-entropy value.
Derive the gradient of softmax cross-entropy with respect to the raw scores.
Read about the Jensen-Shannon divergence and about label smoothing.

Next roadmap ID: 1.15 Eigenvectors and SVD, the directions that matter.

12. Sources and further reading

Information Theory (appendix), Dive into Deep Learning, d2l.ai: definitions of KL divergence, cross-entropy, their properties and the equivalence for training.
Cross-entropy and KL divergence, Eli Bendersky (eli.thegreenplace.net), 2025: bits, entropy, and the five-outcome example.
Eigenvectors and eigenvalues, 3Blue1Brown: a visual lead-in to the next concept, 1.15.
Linear Algebra, Deep Learning book chapter 2, Goodfellow, Bengio and Courville: background for later roadmap items.

Accuracy note: The coin figures (0.152, 3.322, 1.737, 0.737, 0.531) and the classifier loss values come from the video and were re-checked in Python. The weather forecast (example B), the three-picture batch (example C) and the 0.99/0.01 belief (example D) are illustrative numbers chosen for these notes. Forward versus reverse KL behaviour is a common qualitative description, not a theorem for every model; Jensen-Shannon and label smoothing are pointers whose sources I did not open. Example E numbers come from the Greenplace article; the author name and year in that entry come from the site's domain and URL, not from text I could confirm on the page.

Search This Blog