1.13 Entropy: measuring surprise: study notes

Learn AI by Story

1. The idea in one sentence

Entropy is the average surprise of a random outcome, counted in bits: it is low when the outcome is predictable, high when it is uncertain, and it is the lowest average number of yes/no answers (bits) any code needs to describe the outcome.

2. The story from the video

Isha checks the weather forecast. Some news barely surprises her and some is a shock, so can surprise be a number? Rare events carry more surprise: chance 1/2 is 1 bit, 1/8 is 3 bits, 1/64 is 6 bits. Entropy is the average surprise over all outcomes: a fair coin has 1 bit, a coin with 90% heads has 0.47 bits, an always-heads coin has 0. With four weathers (sun 1/2, cloud 1/4, rain 1/8, snow 1/8) the average is 1.75 bits, and the code 0, 10, 110, 111 averages exactly 1.75 bits, which Shannon showed no code can beat.

3. Key definitions

Term	Precise definition	Plain English
Probability, p	A number from 0 to 1 giving how likely an outcome is (see 1.08).	How often it happens, as a fraction.
Logarithm base 2, log2	log2(x) is the power to which 2 is raised to get x; log2(8) = 3.	How many times you can halve something.
Bit	The unit of information; one bit is the information in a fair yes/no answer.	One yes/no question's worth.
Self-information (surprisal)	I(x) = −log2 p(x) bits.	How surprised you should be by one outcome.
Entropy, H	H(X) = −Σ p(x) log2 p(x), the expected self-information.	The average surprise.
Uniform distribution	Every outcome is equally likely (see 1.10).	Complete uncertainty among the options.
Binary entropy	The entropy of a two-outcome variable with probabilities p and 1 − p.	The coin-flip hill.
Code	A rule that maps each outcome to a string of bits.	A secret spelling for each message.
Prefix code	A code in which no codeword is the start of another.	You can read the bits left to right with no separators.
Expected code length	Σ p(x) × (length of the codeword for x).	The average bits per message.
Compression	Re-encoding data to use fewer bits.	Shorter spelling for common things.
Information gain	Entropy before a split minus the weighted entropy after it.	How much a question cuts uncertainty.
Perplexity	2 raised to the entropy (when entropy is in bits).	The effective number of equally likely choices.
Nat	The unit of information when the natural logarithm is used; 1 nat = 1.4427 bits.	The same idea, with a different log.
Differential entropy	The continuous analogue of entropy, defined for continuous variables.	Entropy for a measured quantity like height.

4. Detailed step-by-step explanation

Step 1: Surprise grows as probability shrinks

You hear that the sun rose today. No surprise. You hear that it snowed in a desert. Big surprise. A good measure of surprise should have three properties: a certain event (p = 1) carries zero surprise; the rarer the event, the more surprise; and surprises of independent events add up (the surprise of two independent things happening is the sum of their surprises). Probabilities multiply for independent events, and a logarithm turns multiplication into addition. That is why a logarithm appears.

Step 2: The formula

I(x) = -\log_2 p(x)

The d2l.ai information theory appendix defines self-information exactly this way: for an event with probability p, I = −log2 p bits. Check the video's numbers: p = 1/2 gives −log2(0.5) = 1 bit; p = 1/8 gives 3 bits; p = 1/64 gives 6 bits. The minus sign keeps the answer positive, since log2 of a number below 1 is negative.

Step 3: Why "bits" means yes/no questions

Imagine 8 equally likely outcomes. Each yes/no question can at best halve the options: 8, then 4, then 2, then 1. That takes 3 questions, and −log2(1/8) = 3. So the surprise of an outcome with chance 1/8 equals the number of ideal yes/no questions needed to pin it down. For probabilities that are not powers of 1/2 (such as 0.1), the number of bits is not a whole number (−log2 0.1 = 3.32), which is fine as an average.

Step 4: Entropy is the average surprise

Outcomes happen with different probabilities, so we weight each surprise by how often it happens:

H(X) = -\sum_{x} p(x)\,\log_2 p(x) = \mathbb{E}\left[-\log_2 p(X)\right]

This is the definition used by d2l.ai and by the Greenplace article on cross-entropy: entropy as the expected self-information, measured in bits when the base is 2. In plain words: multiply each outcome's chance by its surprise and add them up. By convention, an outcome with p = 0 contributes 0 to the sum.

Step 5: The coin examples

Fair coin: 0.5 × 1 + 0.5 × 1 = 1 bit.
Coin with P(heads) = 0.9: 0.9 × 0.152 + 0.1 × 3.322 = 0.137 + 0.332 = 0.469 bits.
Coin with P(heads) = 0.99: about 0.081 bits.
Coin that is always heads: 1 × 0 = 0 bits.

The more lopsided the coin, the less there is to learn from a flip.

Step 6: Why the hill peaks in the middle

For any number k of outcomes, entropy is at most log2(k), reached only when all outcomes are equally likely (d2l.ai states this). For two outcomes the maximum is log2(2) = 1 bit at p = 0.5, and the curve falls to 0 at p = 0 and p = 1. Four equally likely outcomes give exactly 2 bits. Maximum uncertainty means maximum entropy.

Step 7: Entropy as a limit on compression

Suppose a source emits sun, cloud, rain or snow with chances 1/2, 1/4, 1/8, 1/8. A naive fixed-length code uses 2 bits per day. A better code gives short words to common outcomes: sun = 0, cloud = 10, rain = 110, snow = 111. The expected length is 0.5 × 1 + 0.25 × 2 + 0.125 × 3 + 0.125 × 3 = 1.75 bits, exactly the entropy. Shannon's source coding result says the entropy is a lower bound on the average bits per symbol of any code that can be decoded unambiguously (the d2l.ai appendix states entropy as this lower bound). So entropy is both a measure of surprise and a hard floor on compression.

5. Everything the video skipped

Where it comes from

Claude Shannon founded information theory. The d2l.ai appendix cites his paper "A Mathematical Theory of Communication" (1948) as the work that introduced entropy, and notes that the word "bit" was coined by John Tukey. Beyond that, these notes do not add historical claims.

Other units: bits, nats and more

The base of the logarithm sets the unit. Base 2 gives bits. The natural logarithm (base e) gives nats, which machine-learning libraries usually use. Converting is a single division: bits = nats ÷ ln 2, so 1 nat = 1.4427 bits. The ideas are identical; only the unit changes.

Entropy depends on the distribution, not on the labels

Renaming sun to "blue" or swapping which outcome gets which probability does not change H. Only the list of probabilities matters.

Properties worth knowing

H is never negative, and H = 0 only when one outcome has probability 1.
H is at most log2(k) for k outcomes, with equality only for the uniform distribution.
Independent sources add: the entropy of two independent variables together is the sum of their entropies.
Merging more outcomes in (adding more equally likely options) raises the maximum entropy by one bit each time the number of options doubles.

Perplexity: entropy as an effective number of choices

Perplexity is 2 raised to the entropy (in bits). Four equally likely outcomes have entropy 2 bits and perplexity 4. The video's weather has entropy 1.75 bits, so perplexity is 2^1.75 = 3.36: as uncertain as a fair die with about 3.4 faces. Language models are often reported with this kind of number: the model is "as unsure as choosing among about that many equally likely words".

Information gain and decision trees

A decision tree asks questions about the data to separate classes. It picks the question that reduces entropy the most: information gain = entropy before the split minus the weighted average entropy of the groups after the split. A split that makes each group pure (all one class) drives entropy to 0.

Joint, conditional, cross-entropy

Entropy of two variables together, entropy of one given the other, and mutual information all build on the same formula. The next concept, 1.14, introduces cross-entropy and KL divergence: the cost of using the wrong distribution to describe the data. Cross-entropy is described in the Greenplace article as the average number of bits needed to encode data from the true distribution P when you assume Q, and it equals the entropy when Q = P.

Continuous variables

For continuous variables the analogue is differential entropy. It behaves differently: it can be negative and it changes when you rescale the variable, so treat it with care.

Limits and trade-offs

Entropy measures uncertainty given a model of the probabilities. If your probabilities are wrong, your entropy number is wrong too.
It does not measure meaning. A random string of letters has high entropy but says nothing.
The compression floor is about the average over many messages with a known distribution; a particular message can be shorter or longer.
Real codes can reach the floor exactly only when probabilities are powers of 1/2, as in the weather example; otherwise they get close.

Links to the roadmap

Earlier: 1.08 probability basics (probabilities that sum to 1), 1.09 Bayes (updating probabilities changes surprise), 1.10 distributions (uniform, Bernoulli), 1.11 mean, variance and correlation (an average of something, just like entropy is an average of surprise), and 1.12 sampling (entropy can be estimated from sample frequencies; more samples, better estimate). Later: 1.14 cross-entropy and KL divergence, then 1.17 vector norms.

6. Worked examples

Example A: entropy of a four-outcome distribution

Probabilities [0.1, 0.5, 0.1, 0.3] (the example used in d2l.ai).

Surprise of each: −log2 0.1 = 3.3219; −log2 0.5 = 1; −log2 0.1 = 3.3219; −log2 0.3 = 1.7370.
Weight by probability: 0.1 × 3.3219 = 0.3322; 0.5 × 1 = 0.5; 0.1 × 3.3219 = 0.3322; 0.3 × 1.7370 = 0.5211.
Add: 0.3322 + 0.5 + 0.3322 + 0.5211 = 1.6855 bits.
Compare with the maximum for 4 outcomes: log2 4 = 2 bits. Our distribution is more predictable than uniform, so it is lower.

Example B: choosing a decision-tree split

(Illustrative data.) Ten emails: 5 spam, 5 not spam. Entropy before the split is 1 bit. Question: does the email contain the word "free"?

Group "yes": 6 emails, 5 spam and 1 not spam. Entropy = −(5/6) log2 (5/6) − (1/6) log2 (1/6) = 0.650 bits.
Group "no": 4 emails, all not spam. Entropy = 0 bits.
Weighted average after the split = (6/10) × 0.650 + (4/10) × 0 = 0.390 bits.
Information gain = 1 − 0.390 = 0.610 bits.

A question that gives a bigger gain than any other would be chosen first.

Example C: how many bits for 1,000 days of weather?

Using the video's weather (1/2, 1/4, 1/8, 1/8):

Fixed 2-bit code: 1,000 × 2 = 2,000 bits.
Prefix code 0, 10, 110, 111: expected 1.75 bits per day, so about 1,750 bits.
Savings: 250 bits, or 12.5%. The real count varies a little from 1,750 because the weather is random, but over many days it averages close to this.
No code can average below 1.75 bits per day for this distribution.

Example D: a lopsided three-way choice (illustrative)

Probabilities [0.7, 0.2, 0.1].

Terms: 0.7 × 0.5146 = 0.3602; 0.2 × 2.3219 = 0.4644; 0.1 × 3.3219 = 0.3322.
H = 0.3602 + 0.4644 + 0.3322 = 1.157 bits.
Uniform maximum = log2 3 = 1.585 bits.
Perplexity = 2^1.157 = 2.23, versus 3 for uniform.

7. Comparison tables and simple diagrams

Entropy of different sources (computed)

Source	Probabilities	Entropy (bits)	Maximum possible (bits)
Always heads	1, 0	0	1
Coin, 99% heads	0.99, 0.01	0.081	1
Coin, 90% heads	0.9, 0.1	0.469	1
Fair coin	0.5, 0.5	1	1
Weather	1/2, 1/4, 1/8, 1/8	1.75	2
Four equal outcomes	1/4 each	2	2
Fair die	1/6 each	2.585	2.585

Surprise by probability

Probability	Surprise (bits)	Yes/no questions to find it among equally likely options
1/2	1	1
1/8	3	3
1/64	6	6
0.9	0.152	about 0.15 on average (not a whole number)
0.1	3.322	not a whole number

Fixed-length code versus prefix code for the weather

Outcome	Chance	Fixed code	Length	Prefix code	Length
Sun	1/2	00	2	0	1
Cloud	1/4	01	2	10	2
Rain	1/8	10	2	110	3
Snow	1/8	11	2	111	3
Average			2.00		1.75

Diagram: the coin entropy hill (text sketch, values computed)

H (bits)
1.0 |            ***
    |         **     **
0.47|       *           (p=0.9 is here)   *
    |    **                                 **
0.0 +----+---------+---------+---------+---> P(heads)
    0   0.1       0.5       0.9        1

The curve is symmetric: a coin with 10% heads has the same entropy as one with 90% heads, 0.469 bits.

8. Common mistakes and myths

Myth: entropy is disorder in a messy room. In these notes it is a precise average of surprise computed from probabilities. (The word is borrowed from physics, but this is the information version.)
Mistake: forgetting the minus sign, which gives negative surprise.
Mistake: mixing log bases. Bits use base 2, nats use base e; compare numbers only in the same unit.
Myth: high entropy means high information content in the sense of meaning. A stream of pure noise has high entropy but carries no meaning.
Mistake: assuming entropy is a property of one message. It is a property of the probability distribution that produces messages.
Mistake: treating entropy as 0 when an outcome has probability near 0. Rare outcomes carry large surprise, but weighted by a tiny probability they add little.
Myth: a lopsided coin has low surprise for every flip. A 99%-heads coin shows tails with a surprise of 6.64 bits, it just happens rarely.
Mistake: believing a clever code can beat entropy on average. For uniquely decodable codes, entropy is a lower bound on the average bits per symbol.
Mistake: thinking the entropy of a sample equals the entropy of the true distribution. With few samples the estimate can be off; see 1.12.

9. Real-life uses

Decision trees: choose the split with the largest information gain (Example B).
Data compression: tools such as zip and image or audio codecs work by giving shorter codes to more common patterns; entropy is the floor they cannot beat on average.
Language models: a model that is very unsure about the next word has high entropy and high perplexity; training pushes uncertainty on real text downward.
Password and key strength: a random choice among 2^40 equally likely options has 40 bits of entropy; this is why secrets are measured in bits.
Weather and spam filters: a forecast of 50% rain is more informative per word than "90% sunny" in terms of what you learn when you find out, because the outcome is more uncertain.
Machine-learning loss: cross-entropy (1.14) is built directly from this formula and is the standard training loss for classifiers.
Experiment design: pick the test whose answer you are most uncertain about, since it has the highest entropy and so can teach the most.

10. Self-check questions

Q: What is the surprise of an event with probability 1/16? A: −log2(1/16) = 4 bits.
Q: What is the entropy of a fair coin? A: 1 bit.
Q: Why is the entropy of a coin with 90% heads less than 1 bit? A: It is usually predictable, so average surprise is only 0.469 bits.
Q: What is the largest possible entropy of a variable with 8 outcomes? A: log2 8 = 3 bits, when all outcomes are equally likely.
Q: Compute the entropy of [1/2, 1/4, 1/4]. A: 0.5 × 1 + 0.25 × 2 + 0.25 × 2 = 1.5 bits.
Q: What is the average length of the code 0, 10, 110, 111 for probabilities 1/2, 1/4, 1/8, 1/8? A: 1.75 bits, equal to the entropy.
Q: Convert 1 nat to bits. A: 1 ÷ ln 2 = 1.4427 bits.
Q: What does information gain measure? A: Entropy before a split minus the weighted entropy after it.
Q: If entropy is 3 bits, what is the perplexity? A: 2^3 = 8 equally likely choices.
Q: Can any decodable code average fewer bits per symbol than the entropy? A: No; entropy is a lower bound on the average code length.

11. Key takeaways, go deeper, and what is next

Surprise of an outcome is −log2 p; rarer means more surprising.
Entropy is the average surprise, −Σ p log2 p, in bits.
It is highest for uniform distributions (log2 k) and zero for certainty.
It sets a floor for compression and drives decision trees, language-model metrics and classification loss.

Go deeper: (1) Huffman coding and how to build optimal prefix codes; (2) conditional entropy and mutual information; (3) the relation of entropy to perplexity in language models; (4) maximum entropy principle; (5) cross-entropy and KL divergence.

Next roadmap ID: 1.14 Cross-entropy and KL divergence.

12. Sources and further reading

Accuracy note: the formulas, the surprise values (1, 3, 6 bits), the coin entropies (0.469 and 0.081 bits), the weather entropy 1.75 bits and the example [0.1, 0.5, 0.1, 0.3] giving 1.6855 bits are computed or taken from the cited sources. The spam email counts in Example B, the [0.7, 0.2, 0.1] distribution and the 1,000-day total are illustrative. Statements about uses (decision trees, compression, language models) are the commonly described ones, not benchmarks.

Entropy: measuring surprise: study notes