1.12 Sampling and the law of large numbers: study notes

Learn AI by Story

1. The idea in one sentence

The more independent random samples you average, the closer the sample average gets to the true average, and the typical error shrinks in proportion to 1 divided by the square root of the number of samples.

2. The story from the video

Dev rolls a die and his first roll is a six. Does that mean the average roll is six? One roll is a sample of size one and it can mislead badly. So he keeps rolling. In one simulated run the running average was 4.1 after 10 rolls, 3.69 after 100, 3.575 after 1,000 and 3.497 after 10,000, settling toward 3.5, which is (1+2+3+4+5+6) ÷ 6. The typical error is 1.71 ÷ √n, so 100 times more data gives 10 times less error. Pollsters and AI engineers rely on exactly this, as long as the sample is random.

3. Key definitions

Term	Precise definition	Plain English
Population	The full set of things (or the full probability distribution) you want to learn about.	Everything you wish you could measure.
Sample	A subset drawn from the population, often written X1, X2, ..., Xn.	The part you actually measure.
Sample size (n)	The number of items in the sample.	How many rolls, people or examples.
Random variable	A quantity whose value is determined by a random process (see 1.08, 1.10).	A number that comes out of a chance experiment.
Expected value (mean), µ	The probability-weighted average of all outcomes, µ = Σ x × P(x).	The long-run average, even if no single outcome equals it.
Variance, σ²	The expected squared distance from the mean (see 1.11).	How spread out one observation is, squared.
Standard deviation, σ	The square root of the variance.	The typical distance of one observation from the mean.
i.i.d.	Independent and identically distributed: each draw is unaffected by the others and comes from the same distribution.	Fair, repeated draws from the same machine.
Sample mean, Mn	Mn = (X1 + ... + Xn) ÷ n.	The average of what you collected.
Law of large numbers (weak form)	For i.i.d. variables with mean µ and any tolerance ε > 0, P(\|Mn − µ\| ≥ ε) → 0 as n → ∞.	With enough data, a big miss becomes very unlikely.
Converges in probability	A sequence converges to a value when the chance of being farther than any tolerance away goes to 0.	It does not promise every run behaves; it promises big misses get rare.
Standard error (SE)	The standard deviation of the sample mean, SE = σ ÷ √n.	The typical error of your average.
Margin of error	A half-width of a confidence interval, about 1.96 × SE for 95% confidence.	The "plus or minus" quoted in a poll.
Bias	A systematic tilt in how the sample is collected or in an estimator.	Being wrong in the same direction every time.
Monte Carlo method	Estimating a quantity by averaging many random draws.	Answer a hard question by simulation.
Central limit theorem (CLT)	For large n, the sample mean is approximately normal with mean µ and variance σ² ÷ n.	Averages look bell-shaped.

4. Detailed step-by-step explanation

Step 1: A sample is a noisy photograph of the population

Suppose a die is fair. The true average (the population mean) is 3.5, because each face has probability 1/6 and (1+2+3+4+5+6) ÷ 6 = 21 ÷ 6 = 3.5. You may not know this in advance; maybe you only have the die. The only way to learn the mean is to sample: roll, record, average.

A single roll is a sample of size one. Its mean is whatever face came up: it could be 1, 6, or anything between. A six tells you almost nothing, because a fair die shows a six one time in six.

Step 2: Averages calm down as you add data

Each new roll nudges the running average. When n is small, one roll moves it a lot: going from 5 rolls to 6 rolls changes the weight of a new roll to 1/6. When n is 1,000, one new roll changes the average by at most about 5 ÷ 1,000 = 0.005. The running average therefore wobbles wildly at the start and then drifts more and more slowly. That is the "settling" picture in the video.

Step 3: Why the average improves (the mechanism)

There are two facts behind the law.

The sample mean is unbiased: its expected value is µ. On average it aims at the right target.
Its variance shrinks: because independent draws partly cancel each other (a high roll is usually offset by a low roll), Var(Mn) = σ² ÷ n.

The second fact is the heart of it. Written as a formula:

\operatorname{Var}(M_n) = \frac{\sigma^2}{n}, \qquad \mathrm{SE} = \frac{\sigma}{\sqrt{n}}

In words: add up n independent things and their variances add (n × σ²). Divide the sum by n and the variance gets divided by n², leaving σ² ÷ n. Taking the square root gives the standard error σ ÷ √n.

Step 4: From a small variance to a guarantee (Chebyshev)

Chebyshev's inequality says that for any random variable, the chance of landing more than ε from its mean is at most variance ÷ ε². Applied to the sample mean:

P\left(|M_n - \mu| \ge \varepsilon\right) \le \frac{\sigma^2}{n\,\varepsilon^2}

As n grows the right-hand side goes to 0, so the left-hand side must as well. That is the weak law of large numbers, in the form given in standard lecture notes such as Purdue ECE 302 and Brown CS145 (see Sources).

Step 5: How fast? The one over root n rule

The die has σ = 1.7078 (variance 2.9167). So the standard error is 1.7078 ÷ √n:

n	Standard error (computed)
10	0.540
100	0.171
1,000	0.054
10,000	0.0171

To cut the error by 10, you need 100 times the data. This square-root slowness is the price of randomness: the first few samples teach you a lot, later ones teach you less and less.

Step 6: Honest limits

The law promises the average approaches the truth. It does not promise that the count of sixes evens out, that a streak must reverse, or that your sample is representative if it was collected badly. Those are covered in sections 5 and 8.

5. Everything the video skipped

Weak versus strong law

The video describes the weak law: for each fixed tolerance, the probability of a big miss goes to 0. There is also a strong law, which says that with probability 1 the actual running average of a single endless run converges to µ. The weak law is about probabilities at each n; the strong law is about the whole path. For beginners the practical message is the same: more random data, closer average.

What the law needs

Independence (or at least weak dependence). If the samples copy each other, they do not cancel. Ten thousand copies of one roll is still one roll's worth of information.
A finite mean. The standard counterexample is the Cauchy distribution, which has no mean: its sample average does not settle down, no matter how large n is. This is a well-known textbook fact. Heavy-tailed data in real life can behave in a similar, less extreme way, where a rare giant value keeps jerking the average.
Representative sampling. The law says the average of your sampling process converges to the mean of that process. If the process is tilted (for example, polling only people who answer a landline), it converges confidently to the wrong number.

The central limit theorem: what comes next

The law of large numbers tells you where the average goes. The central limit theorem (CLT) tells you the shape of the leftover error: for large n, the sample mean is approximately normal with mean µ and variance σ² ÷ n (see the Brown lecture notes in Sources). That is why a 95% margin of error is about 1.96 standard errors: for a normal curve, about 95% of the probability lies within 1.96 standard deviations of the center (see 1.10 distributions). The video does not cover the CLT, so treat it as the natural sequel.

Sampling methods

Method	How it works	Typical use
Simple random sample	Every member has the same chance of being picked.	The textbook baseline for polls.
Stratified sample	Split the population into groups, then sample within each group.	Make sure small groups are represented.
Cluster sample	Pick whole groups (schools, villages), then survey everyone inside.	When a full list of individuals is unavailable.
Convenience sample	Whoever is easy to reach.	Quick and cheap, but risky because it is often biased.
Random mini-batch	Shuffle a data set and take n examples at a time.	Training neural networks.

Sampling without replacement

When you draw from a small, finite population without putting items back, the draws are slightly negatively correlated and the error is a little smaller than σ ÷ √n. For a sample that is a tiny fraction of a huge population the correction is negligible, which is why a poll of 1,000 works the same for a city or a country.

Trade-offs and limits

Diminishing returns: halving the error costs four times the data.
Cost: each extra sample may cost money (a survey) or compute (a simulation or a labeled example).
Variance versus bias: more data shrinks random error (variance) but does nothing to systematic error (bias).

Links to the roadmap

Earlier: 1.08 probability basics (random variables and expectation), 1.10 distributions (the normal curve behind margins of error), 1.11 mean, variance and correlation (the spread σ that sets the error), and 1.07 the chain rule, which you meet again when a mini-batch gradient is a sample average of per-example gradients. Later: 1.13 entropy next, and 1.17 vector norms, where averages and distances in many dimensions appear again.

6. Worked examples

Example A: how many rolls for an average good to within 0.1?

Goal: a 95% margin of error of 0.1 on the average of a fair die.

Spread of one roll: σ = 1.7078.
Margin = 1.96 × σ ÷ √n. Set it to 0.1.
√n = 1.96 × 1.7078 ÷ 0.1 = 33.47, so n = 33.47², about 1,120 rolls.
Check: with n = 1,120 the standard error is 1.7078 ÷ 33.47 = 0.051, and 1.96 × 0.051 = 0.100.

With n = 100 the same 95% margin is 1.96 × 0.1708 = 0.335. So the video's run that read 3.69 after 100 rolls (off by 0.19) is well within what chance allows. This uses the normal approximation from the CLT, so it is approximate.

Example B: the poll

A pollster asks n = 1,000 randomly chosen people a yes/no question. Suppose the true share is p = 0.5.

A yes/no answer (1 or 0) has variance p(1 − p) = 0.25, so σ = 0.5.
SE = 0.5 ÷ √1,000 = 0.5 ÷ 31.62 = 0.0158, about 1.6 percentage points.
95% margin = 1.96 × 0.0158 = 0.0310, about 3.1 points. That is the "plus or minus 3" in the video.
For a margin of 1 point instead: n = (1.96 × 0.5 ÷ 0.01)² = 98² = 9,604 people, nearly ten times as many.

This assumes a simple random sample. The Brown notes show a more conservative Chebyshev-based figure of 50,000 for the same 1-point goal because Chebyshev holds for any distribution and so is looser.

Example C: the Chebyshev bound (illustrative numbers)

Take σ² = 1 and tolerance ε = 0.1. The bound is σ² ÷ (nε²) = 1 ÷ (n × 0.01) = 100 ÷ n.

n = 100: bound = 1.0, which says nothing (a probability is at most 1 anyway).
n = 10,000: bound = 0.01, so at most a 1% chance of missing by more than 0.1.

The bound is proven but loose; real behaviour is usually much better. These values of σ and ε are illustrative.

Example D: reading the video's simulation

The video's run (one simulation, numpy default_rng(7); a different seed gives different values) reported 4.1, 3.69, 3.575, 3.497 at n = 10, 100, 1,000, 10,000. Distances from 3.5: 0.6, 0.19, 0.075, 0.003. Standard errors: 0.540, 0.171, 0.054, 0.017. Every miss is within about 1.4 standard errors, which is typical and consistent with theory. Note the misses do not shrink perfectly smoothly: that is chance, not a failure of the law.

7. Comparison tables and simple diagrams

Small sample versus large sample

	Small sample (n = 10)	Large sample (n = 10,000)
Standard error for a die (computed)	0.540	0.0171
Typical look of the running mean	Jumpy, can be far off	Smooth, hugging 3.5
Cost	Cheap	1,000 times more rolls
Fixes a biased sampling method?	No	No

Weak law, strong law, CLT at a glance

Result	What it says	Question it answers
Weak law of large numbers	Big misses become unlikely as n grows.	Does the average approach the truth?
Strong law	With probability 1, the whole running average converges.	Does one endless run converge?
Central limit theorem	The error is roughly normal with spread σ ÷ √n.	How big and what shape is the error?
Chebyshev bound	Probability of a miss is at most σ² ÷ (nε²).	A guaranteed but loose bound for any distribution.

Diagram: error versus data (text sketch)

error
 0.54 |*
      |
 0.17 |   *
 0.05 |        *
 0.017|              *
      +----+----+-----+----> n
         10  100  1,000 10,000

Each step to the right multiplies the data by 10 and divides the error by about 3.16 (the square root of 10).

8. Common mistakes and myths

Myth: the law of large numbers means things must even out. If a coin shows five heads in a row, the next flip is still 50-50. This belief is the gambler's fallacy. The law says the average approaches 0.5 because early flukes get diluted by many new flips, not because tails are "due".
Myth: a bigger sample fixes a biased sample. A million responses from a self-selected online poll is still self-selected.
Mistake: confusing the sum with the average. The raw count of sixes minus n/6 typically grows (on the order of √n); it is the fraction of sixes that settles.
Mistake: thinking one-over-root-n means a small improvement is cheap. Halving the error needs 4 times the data.
Mistake: applying the law to non-independent data, such as consecutive days of a time series, without thought.
Myth: the population size matters a lot. For a small fraction of a large population, the sample size n drives the error, not the population size.
Mistake: reading a margin of error as a hard limit. A 95% interval misses about 1 time in 20 by design.
Mistake: ignoring heavy tails, where a few extreme values keep dominating the average.

9. Real-life uses

Polling: 1,000 randomly chosen respondents give about ±3.1 points at 95% confidence (Example B).
Mini-batch training: a batch of 32 examples gives a noisy but unbiased estimate of the gradient over the whole data set. Bigger batches reduce the noise by about √ (batch size), but with diminishing returns.
Test sets: the accuracy on a held-out test set of 10,000 examples estimates real-world accuracy. If the model is right 90% of the time (illustrative), SE = √(0.9 × 0.1 ÷ 10,000) = 0.003, about 0.3 points.
Monte Carlo simulation: estimate an average, an area or a risk by averaging many random draws; ten thousand draws give about one hundredth of one draw's spread.
Quality control: inspect a random sample of items from a batch to estimate the defect rate.
A/B tests: compare average outcomes of two random groups of users.
Insurance: pooling many independent risks makes the average claim predictable, even though each claim is not.

10. Self-check questions

Q: What is the expected value of a fair six-sided die? A: (1+2+3+4+5+6) ÷ 6 = 3.5.
Q: State the weak law of large numbers in plain words. A: For i.i.d. samples with a finite mean, the chance that the sample mean is farther than any chosen tolerance from the true mean goes to 0 as n grows.
Q: What is the standard error of the mean? A: σ ÷ √n, the standard deviation of the sample mean.
Q: If you want 5 times less error, how much more data do you need? A: 25 times, because error ∝ 1 ÷ √n.
Q: A poll of 1,000 random people has what 95% margin of error at p = 0.5? A: about 3.1 percentage points.
Q: Why does a biased sample not improve with size? A: Averaging shrinks random error only; the bias is a systematic shift that is the same in every sample.
Q: Is a coin that landed heads five times in a row more likely to land tails? A: No. Independent flips have no memory; thinking otherwise is the gambler's fallacy.
Q: What assumption does the Cauchy distribution break? A: It has no finite mean, so the law of large numbers does not apply to its sample average.
Q: What does Chebyshev's inequality give for the sample mean? A: P(|Mn − µ| ≥ ε) ≤ σ² ÷ (nε²), a guaranteed but loose bound.
Q: Where does AI use sampling averages? A: Mini-batch gradient estimates, test-set accuracy and Monte Carlo methods.

11. Key takeaways, go deeper, and what is next

Averages of many independent random samples approach the true mean.
Typical error is σ ÷ √n: 100 times more data, 10 times less error.
Samples must be random and independent; size cannot cure bias.
The law is about averages, not about streaks or sums.

Go deeper: (1) the central limit theorem and confidence intervals; (2) the strong law; (3) bootstrap resampling; (4) stratified sampling and importance sampling in Monte Carlo; (5) how batch size affects training noise.

Next roadmap ID: 1.13 Entropy: measuring surprise.

12. Sources and further reading

Accuracy note: the theorems, the die mean 3.5 and σ = 1.7078, the standard errors and the poll margin of about 3.1 points are computed or standard results. The running means 4.1, 3.69, 3.575 and 3.497 come from one simulated run (a different seed gives different values). The 90% test accuracy and the σ² = 1, ε = 0.1 Chebyshev case are illustrative. The Cauchy example is a standard textbook fact that I did not re-derive from the sources above.

Sampling and the law of large numbers: study notes