1.39 The central limit theorem: study notes

Learn AI by Story

1. The idea in one sentence

Add up (or average) many small, independent random things and the result follows a bell curve, almost no matter what shape the individual things had. Plain English: the crowd's average is predictable even when each person is not.

2. The story from the video

Divya runs a dice game at the school fair. One fair die is flat: each face 1 to 6 has probability 1/6, so there is no "typical" roll. The mean is 3.5, the variance is 35/12 ≈ 2.917 and the standard deviation is about 1.708.

Then she lets players roll two dice and add them. Suddenly the totals are not flat. There are 36 equally likely pairs, and the number of ways to get each sum 2, 3, ..., 12 is 1, 2, 3, 4, 5, 6, 5, 4, 3, 2, 1. A total of 7 has 6 ways, a total of 2 has only 1. The flat shape has already become a hill.

Divya keeps going. She averages 4 dice, then 25, then 100. Each time the hill gets narrower and rounder. The spread of the average of n dice is 1.708/√n: for n = 4 it is 0.854, for n = 25 it is 0.342, and for n = 100 it is 0.171. For the average of 25 dice, about 95% of the time the result lies in 3.5 ± 1.96 × 0.342, that is from 2.83 to 4.17. A simulation of 200,000 such averages found 95.4% inside that range and a standard deviation of 0.342. The lesson of the video: the data (one roll) stays flat, but the average becomes normal.

3. Key definitions (glossary)

Random variable: a number whose value depends on chance, such as the face of a die. Plain English: a quantity you cannot predict exactly.
Population mean μ: the long-run average value E[X]. Plain English: the centre the data cluster around.
Variance σ² and standard deviation σ: σ² = E[(X − μ)²] and σ is its square root. Plain English: how far a typical value strays from the mean.
i.i.d.: independent and identically distributed, meaning each draw comes from the same distribution and does not influence the others. Plain English: repeated fair draws from the same bag.
Sample mean X̄: X̄ = (X₁ + X₂ + ... + Xₙ)/n. Plain English: the average of what you actually saw.
Sampling distribution: the distribution of a statistic (like X̄) over all possible samples of the same size. Plain English: what the average would look like if you repeated the whole experiment many times.
Standard error (SE): the standard deviation of the sampling distribution, SE = σ/√n for the sample mean. Plain English: how much the average wobbles from sample to sample.
Normal (Gaussian) distribution N(μ, σ²): the bell curve with mean μ and variance σ². Plain English: the familiar symmetric hill.
z-score: z = (x − μ)/σ, the number of standard deviations from the mean. Plain English: a result expressed in "how many wobbles away".
Central limit theorem (CLT): for i.i.d. X₁, ..., Xₙ with finite mean μ and finite variance σ², the standardised average (X̄ − μ)/(σ/√n) tends to N(0, 1) as n grows. Plain English: averages of many things look normal.
Law of large numbers (LLN): X̄ converges to μ as n grows. Plain English: the average settles down on the truth.
Heavy-tailed distribution: one where extreme values are so likely that the variance can be infinite (the Cauchy distribution is the classic case). Plain English: wild outliers keep showing up.

4. Detailed explanation, from basics to the full picture

Step 1: One die is flat

For a fair die, P(X = k) = 1/6 for k = 1, ..., 6. The mean is E[X] = (1 + 2 + ... + 6)/6 = 3.5. For the variance, E[X²] = (1 + 4 + 9 + 16 + 25 + 36)/6 = 91/6 ≈ 15.167, so Var(X) = 91/6 − 3.5² = 35/12 ≈ 2.917, and σ ≈ 1.708. The shape is a flat bar chart with six equal bars. Nothing about it looks like a bell.

Step 2: Two dice make a triangle

Counting pairs gives the ways 1, 2, 3, 4, 5, 6, 5, 4, 3, 2, 1 for sums 2 to 12, out of 36. The middle is favoured simply because there are more ways to reach it. Extreme sums need both dice to be extreme, which is rarer. This "middle has more combinations" idea is the whole engine of the CLT.

Step 3: Means add, variances add

For independent variables, E[X₁ + X₂ + ... + Xₙ] = nμ and Var(X₁ + ... + Xₙ) = nσ². Variances add, standard deviations do not. So the sum has standard deviation σ√n, which grows slowly (like a square root), not like n.

Step 4: The average shrinks like 1/√n

Dividing the sum by n gives the average X̄. Dividing by n divides the standard deviation by n and the variance by n²: Var(X̄) = nσ²/n² = σ²/n. So SE = σ/√n. For dice, 1.708/√25 = 0.342. The average has the same centre μ but is squeezed tighter. To halve the wobble you need four times as much data; to cut it to one tenth you need 100 times as much.

Step 5: The shape becomes normal

The CLT says that, besides the centre and width being known, the shape of X̄ approaches N(μ, σ²/n). Written in standardised form:

(X̄ − μ)/(σ/√n) → N(0, 1) as n → ∞.

This holds whether the original data were flat (dice), two-humped, skewed or discrete, as long as the draws are independent with finite variance. Each draw adds a small push up or down; many small pushes of random sign mostly cancel, and the leftover has the bell shape.

Step 6: Using the bell curve

For a normal curve, about 68% of the mass is within 1 standard deviation, 95% within 1.96 (often rounded to 2), and 99.7% within 3. So the average of n draws lies within μ ± 1.96 σ/√n about 95% of the time. For 25 dice: 3.5 ± 1.96 × 0.342 = 3.5 ± 0.67, i.e. 2.83 to 4.17.

Step 7: Why it matters for the rest of statistics

If we want to know how far a sample average might be from the truth, we need its sampling distribution. The CLT hands us a normal one without knowing the data's shape. That is why confidence intervals (video 1.41) and hypothesis tests (video 1.40) can use the normal curve for many different kinds of data.

5. Everything the video skipped

Formulas in simple words

Sample mean: X̄ = (1/n) ∑ Xᵢ. Add all values and divide by how many.
Standard error: SE = σ/√n. The spread of the data divided by the square root of the sample size.
Standardising: Z = (X̄ − μ)/(σ/√n). Subtract the centre and divide by the wobble.
Sum version: ∑ Xᵢ is approximately N(nμ, nσ²).
Binomial as a CLT case: a Binomial(n, p) count is a sum of n Bernoulli(p) draws, so it is approximately N(np, np(1 − p)). This is why the binomial in [D2L-DIST] looks Gaussian when n = 10, 100, 1000.
When σ is unknown we replace it by the sample standard deviation s. For small n and roughly normal data, the Student's t distribution corrects for that extra uncertainty.

Conditions

Independence: the draws must not push each other. Strongly dependent data (for example neighbouring days of a stock price, or pixels of one image) shrink the effective sample size, and the 1/√n rule is too optimistic.
Identical distribution (classic version): there are more general versions (Lindeberg, Lyapunov) that allow different distributions if no single term dominates.
Finite variance: this is the condition people forget. The Cauchy distribution has no finite variance. Averaging Cauchy draws does not help: the average of n Cauchy draws is itself Cauchy, as wild as a single draw. In a quick simulation of one million Cauchy draws the running mean came out as about 0.47 and would jump to a different value with a different seed, never settling.
"n = 30" is a rule of thumb, not a theorem. For mildly skewed data it is often fine. In a quick simulation, averages of 30 draws from a strongly skewed exponential distribution (mean 1, σ = 1) landed inside 1 ± 1.96/√30 about 95.3% of the time, close to 95%. For extremely skewed data, such as rare events, hundreds or thousands of draws may be needed.

Limitations

The CLT is about the average (or sum), not about the data. A histogram of raw dice rolls stays flat forever.
It describes the middle of the distribution well and the far tails poorly. "Six sigma" events in averages of heavy-tailed data are more likely than the normal curve says.
It is a limit statement. How close n is to "enough" is answered by the Berry-Esseen theorem (background): the error shrinks like 1/√n and is bigger for skewed data.
The LLN and the CLT are different: the LLN says where the average goes, the CLT says how it wobbles on the way.

Background

Pierre-Simon Laplace gave the central limit theorem in its general form, as noted in [PRML] section 2.3. Abraham de Moivre had earlier shown the special case for coin flips (background). Bishop's Fig. 2.6 shows averages of N = 1, 2, 10 uniform variables turning into a bell.
The name "central" refers to its central importance in probability, not to the centre of the curve.

Links on the roadmap

1.34 Joint, marginal and independent: independence is the key assumption here.
1.35 Expectation and covariance: E[X], Var(X) and the rule that variances of independent terms add.
1.36 Covariance matrix and multivariate Gaussian: there is a multivariate CLT, and sums of correlated vectors become multivariate Gaussians.
1.40 Hypothesis testing and p-values and 1.41 Confidence intervals and bootstrapping use the CLT directly.
1.43 Monte Carlo sampling: the error of a Monte Carlo estimate shrinks like σ/√n for exactly this reason.

6. Worked examples with real numbers

Example A: Divya's average of 25 dice

Setup: roll 25 fair dice and take the average. Where does it land?

One die: μ = 3.5 and σ² = 35/12 ≈ 2.917.
Standard deviation of one die: σ = √2.917 ≈ 1.708.
Standard error for n = 25: SE = 1.708/√25 = 1.708/5 = 0.342.
95% range: 3.5 ± 1.96 × 0.342 = 3.5 ± 0.670.
Lower end: 3.5 − 0.670 = 2.83. Upper end: 3.5 + 0.670 = 4.17.
Check by simulation: 200,000 averages gave 95.4% inside, with standard deviation 0.342.

Result: the average of 25 dice lands between about 2.83 and 4.17 about 95% of the time.

Example B: How likely is a high average of 100 dice?

Setup: a stall owner claims a player's 100 rolls averaged 3.7 or more. Is that unusual for fair dice?

SE for n = 100: 1.708/√100 = 0.1708.
Distance from the mean: 3.7 − 3.5 = 0.2.
z-score: z = 0.2/0.1708 ≈ 1.171.
One-sided probability from the normal curve: P(X̄ ≥ 3.7) ≈ 0.121.
Two-sided probability of being at least this far from 3.5 in either direction: 2 × 0.121 ≈ 0.242.

Result: about a 12% chance of an average of 3.7 or higher with fair dice. That is not unusual, so no evidence of a loaded die.

Example C: How many dice for a narrow answer?

Setup: we want the average to be within ±0.1 of 3.5 about 95% of the time.

We need 1.96 × σ/√n ≤ 0.1.
Rearrange: √n ≥ 1.96 × 1.708/0.1 = 33.47.
Square: n ≥ 33.47² ≈ 1,120.5, so n = 1,121 dice.
Cross-check: SE = 1.708/√1121 = 0.0510 and 1.96 × 0.0510 ≈ 0.100.

Result: about 1,121 dice. Going from ±0.67 (25 dice) to ±0.1 is about 6.7 times more precise, and it costs about 45 times more dice (1,121 ÷ 25), the square-root law at work.

Example D: Noise in mini-batch gradients

Setup: suppose each training example gives a gradient component with standard deviation σ = 2.0 around its true average (illustrative number). A mini-batch averages B examples.

Mini-batch noise: SE = σ/√B.
B = 64: 2.0/8 = 0.25.
B = 256: 2.0/16 = 0.125.

Result: quadrupling the batch from 64 to 256 halves the gradient noise. This is a CLT-style argument (background reasoning), which is why very large batches give diminishing returns.

7. Comparison tables

Data versus average

Item	One draw	Average of n draws
Centre	μ	μ
Spread	σ	σ/√n
Shape	Whatever it was	Approximately normal for large n
Example (dice)	Flat, sd 1.708	n = 25: sd 0.342

Standard error of the average of dice

n	SE = 1.708/√n	95% half-width (1.96 × SE)
1	1.708	3.35
4	0.854	1.67
25	0.342	0.67
100	0.171	0.33

LLN versus CLT

Theorem	Says	Needs
Law of large numbers	X̄ → μ	Finite mean
Central limit theorem	X̄ is approximately N(μ, σ²/n)	Independence, finite variance
Both fail for	Cauchy (no mean, no variance)	-

8. Common mistakes and myths

Myth: the data become normal. No. Only the average (or sum) does. A pile of raw dice rolls stays flat however large it gets.
Myth: n = 30 always works. It is a rule of thumb. Skewed or heavy-tailed data need more.
Mistake: forgetting the square root. The wobble shrinks like 1/√n, not 1/n. Ten times the data gives only about 3.2 times the precision.
Mistake: using it for dependent data. Autocorrelated data (time series, repeated measurements of the same person) have fewer effective samples than rows.
Mistake: ignoring infinite variance. With Cauchy-like heavy tails, averaging does not rescue you.
Myth: the CLT says the sample mean is the true mean. It says the sample mean is likely close, with a known typical error. That is not the same as being right.
Mistake: confusing σ and σ/√n. σ describes single data points, σ/√n describes averages. Using σ for an average gives intervals that are far too wide.
Mistake: gambler's fallacy. The CLT does not say a run of high dice will be "corrected" by low ones. Future rolls stay fair; the early bump is simply diluted by the growing n.

9. Real-life and AI uses

Error bars on accuracy. A test accuracy is an average of 0/1 outcomes, so by the CLT it has standard error √(p̂(1 − p̂)/n). [GF5] section 5.4.4 uses this to build a 95% interval μ̂ ± 1.96 SE and says algorithm A is better than B if the upper bound of A's interval for error is below the lower bound of B's.
Mini-batch training. The mini-batch gradient is an average, so its noise has standard deviation roughly proportional to 1/√B (background reasoning from the CLT).
Opinion polls and quality control. Polls of 1,000 people give about ±3 points because of σ/√n. Factories average measurements to detect drift.
Averaging measurements. Averaging repeated readings of a noisy sensor cuts the error by √n.
Monte Carlo methods. Estimating an expectation by averaging random samples has error shrinking like 1/√n (video 1.43).
A/B tests. Differences in conversion rates are tested by normal approximations, built on the CLT (video 1.40).
Why Gaussians are everywhere. Measurement noise is the sum of many small independent causes, so Gaussian noise is a natural model. This also motivates the Gaussian-noise assumption that turns MSE into maximum likelihood [GF5] section 5.5.1.

10. Self-check questions (with answers)

What are the mean and standard deviation of one fair die? The mean is 3.5 and the standard deviation is √(35/12) ≈ 1.708.
In how many ways can two dice sum to 7, and what is the probability? Six ways out of 36, so 6/36 = 1/6.
What is the standard error of the average of 100 dice? 1.708/√100 ≈ 0.171.
What does the CLT say becomes normal, the data or the average? The average (or sum) of many independent draws, not the data themselves.
If you want to cut the standard error to one third, how much more data do you need? Nine times as much, because SE ∝ 1/√n.
Give the 95% range for the average of 25 dice. 3.5 ± 1.96 × 0.342, so about 2.83 to 4.17.
Why does the CLT fail for the Cauchy distribution? It has no finite variance (and no mean), so averages are as wild as single draws.
How is a Binomial(n, p) count related to the CLT? It is a sum of n Bernoulli(p) draws, so it is approximately N(np, np(1 − p)) for large n.
Why is a mini-batch of 256 less noisy than one of 64? Gradient noise ∝ 1/√B, so quadrupling B halves it.
Does the CLT justify the claim "after 10 heads in a row, tails is due"? No. Draws are independent. The CLT only says the average of many draws gets close to the mean because early deviations are diluted.

11. Key takeaways, go deeper, next

Key takeaways

The mean of X̄ is μ, its standard deviation is σ/√n, and for large n its shape is approximately normal.
The data keep their shape. Only the average becomes bell-shaped.
Need independence and finite variance. Cauchy breaks the theorem.
Precision improves like √n, so each extra digit of accuracy costs 100 times more data.
This one theorem underlies error bars, confidence intervals, p-values and Monte Carlo error.

Go deeper

Simulate averages of other distributions (exponential, uniform, a coin) with numpy and compare their histograms with the normal curve.
Look at Bishop's Fig. 2.6 and the D2L binomial-to-Gaussian plots for n = 1, 10, 100, 1000.
Read about the Berry-Esseen bound (background) to see how fast the approximation improves.

Next on the roadmap: 1.40 Hypothesis testing and p-values.

12. Sources and further reading

Zhang, Lipton, Li, Smola, Dive into Deep Learning (online book), 22.8 Distributions
Bishop, Pattern Recognition and Machine Learning (2006), 2.3 The Gaussian distribution (PDF)
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 5, Machine Learning Basics, section 5.4.4 Standard error
Zhang, Lipton, Li, Smola, Dive into Deep Learning (online book), 22.10 Statistics
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 3, Probability and Information Theory

Accuracy note: The dice numbers (mean 3.5, variance 35/12, standard errors 0.854, 0.342, 0.171, the interval 2.83 to 4.17 and the 95.4% simulation result) were computed and checked in Python. The mini-batch noise value σ = 2.0 and the player's average of 3.7 are illustrative. The facts that Laplace gave the CLT and that PRML shows averages of N = 1, 2, 10 uniforms come from the cited book; de Moivre's earlier coin result, the Berry-Esseen theorem, the Cauchy simulation and the mini-batch 1/√B argument are background knowledge, not taken from the sources.

The central limit theorem: study notes