1.41 Confidence intervals and bootstrapping: study notes

Learn AI by Story

1. The idea in one sentence

A single number measured on a sample (like "85% accuracy on 100 test images") is only a noisy estimate, so we report a range around it, a confidence interval, and we can build that range either with a formula (p̂ ± 1.96 × SE) or by re-sampling our own data thousands of times (the bootstrap).

2. The story from the video

Rhea trains a cat-versus-dog classifier. She tests it on 100 photos and it gets 85 right, so p̂ = 85/100 = 0.85. Her friend asks: "Is it really 85% good, or did you just get a lucky batch of photos?" Rhea realises that a different set of 100 photos would give a different score, maybe 82, maybe 89. The 0.85 is one draw from a cloud of possible scores.

She first uses the textbook shortcut. The standard error is SE = √(0.85 × 0.15 / 100) = 0.0357. The 95% interval is 0.85 ± 1.96 × 0.0357 = 0.85 ± 0.070, so 0.780 to 0.920. She reports "85%, 95% CI 78% to 92%".

Then she tries the bootstrap, an idea from Bradley Efron (1979). She treats her 100 results (85 ones and 15 zeros) as a small stand-in for the world. She draws 100 results with replacement, computes the accuracy, and repeats this 10,000 times. The middle 95% of those 10,000 accuracies runs from 0.78 to 0.92, and their standard deviation is 0.0356. The shortcut and the computer agree. A toy picture from the video: the sample 3, 5, 7, 8, 12 has mean 7.0; one resample 5, 5, 8, 12, 3 has mean 6.6; another, 7, 7, 12, 12, 8, has mean 9.2. Same data, shuffled with repeats, different means. That spread is the uncertainty.

Finally Rhea learns that more photos shrink the interval: ±7.0 points at n = 100, ±3.5 at 400, ±2.2 at 1,000 and ±0.7 at 10,000. Quadrupling the data only halves the width.

3. Key definitions (glossary)

Population parameter: the true, fixed but unknown quantity we care about, such as the classifier's true accuracy p on all photos it will ever see. Plain English: the real answer we can never fully measure.
Estimator and estimate: an estimator is a rule that turns data into a guess (p̂ = correct ÷ n); the estimate is the number it produces on one sample (0.85). Plain English: the recipe versus the one dish you cooked.
Sampling distribution: the distribution of an estimate over all the samples we could have drawn. Plain English: the cloud of scores you would see if you re-ran the test on fresh photos again and again.
Standard error (SE): the standard deviation of the sampling distribution of an estimator. For an accuracy it is √(p̂(1 − p̂)/n); for a mean it is σ/√n. Plain English: how far a typical re-run score lands from the average.
Confidence interval (CI): a range [L, U] computed from the data by a procedure that, over repeated samples, contains the true parameter a stated fraction (such as 95%) of the time. Plain English: a range built by a method that is right 95 times in 100.
Confidence level: the long-run coverage of the procedure, written 1 − α (α = 0.05 gives 95%). Plain English: how often the method works, not the chance for this one interval.
Critical value z*: the multiplier from the normal curve that leaves α/2 in each tail; z* = 1.96 for 95%. Plain English: roughly "two standard errors".
Margin of error: the half-width of the interval, z* × SE (0.070 for Rhea). Plain English: the "plus or minus".
Wald (normal) interval: p̂ ± z* √(p̂(1 − p̂)/n), justified by the central limit theorem. Plain English: the quick formula everyone learns first.
Wilson interval: an improved interval for proportions that re-centres and re-sizes the Wald interval. Plain English: the formula that behaves when n is small or p̂ is near 0 or 1.
Student's t-interval: x̄ ± t* s/√n, using the t distribution with n − 1 degrees of freedom. Plain English: the honest version of the mean interval when σ is estimated from only a few points.
Bootstrap: estimate the sampling distribution by drawing many samples of size n with replacement from the observed data. Plain English: pull yourself up by your own data.
Bootstrap replicate: the statistic computed on one resample. Plain English: one pretend re-run of the experiment.
Percentile interval: the 2.5th and 97.5th percentiles of the bootstrap replicates. Plain English: chop 2.5% off each end of the pile of replicates.
Out-of-bag (OOB): the items not drawn into a given bootstrap sample, about 1/e ≈ 36.8% of them. Plain English: the leftovers you can test on for free.

4. Detailed explanation, from basics to the full picture

Step 1: Why one number is not enough

Your test set is a random sample from all the photos the model will meet. The true accuracy p is a fixed number. The measured p̂ wobbles from sample to sample. If you report only "85%", the reader cannot tell whether you tested 20 photos or 20 million. A confidence interval puts the wobble on the page.

Step 2: The size of the wobble (standard error)

Each photo is a coin flip: correct (1) with probability p, wrong (0) otherwise. One flip has variance p(1 − p). The average of n independent flips has variance p(1 − p)/n, so

SE = √(p(1 − p)/n).

We do not know p, so we plug in p̂. For Rhea: √(0.85 × 0.15 / 100) = √0.001275 = 0.0357. The √n in the denominator is the key: the wobble shrinks like 1/√n.

Step 3: From SE to an interval (the central limit theorem)

The central limit theorem (video 1.39) says that an average of many independent measurements is close to normal. About 95% of a normal curve lies within 1.96 standard deviations of its centre. So the estimate p̂ lands within 1.96 SE of the true p about 95% of the time, and the interval p̂ ± 1.96 SE therefore catches p about 95% of the time. Goodfellow, Bengio and Courville give this exact recipe: μ̂ ± 1.96 SE.

Step 4: What "95% confident" really means

This is the most misunderstood sentence in statistics. The true p is fixed. The interval is the random thing, because it moves with the sample. "95%" describes the procedure: if you repeated the whole experiment many times and built an interval each time, about 95% of those intervals would contain p. It is not a statement that this particular interval has a 95% chance of containing p. Once computed, it either does or does not. The Dive into Deep Learning chapter on statistics (following Jerzy Neyman, who introduced confidence intervals in 1937) states it this way and lists three classic fallacies drawn from Morey et al. (2016), covered in section 8.

Step 5: How n controls the width

Width is proportional to 1/√n. With p̂ = 0.85 the half-width 1.96 × √(0.1275/n) is 7.0 points at n = 100, 3.5 at n = 400, 2.2 at n = 1,000 and 0.7 at n = 10,000. To halve the error you need four times the data; to cut it by 10 you need 100 times the data. This is why chasing the last decimal of accuracy on a small test set is meaningless.

Step 6: The bootstrap idea

The formula needed theory (a CLT and a neat SE). For many statistics, such as median, F1 score, AUC or the ratio of two numbers, no clean formula exists. The bootstrap replaces mathematics with computing. Your sample is your best picture of the population, so resample from it:

Draw n items from your n data points with replacement (some appear twice, some not at all).
Compute the statistic on this resample.
Repeat B times (1,000 to 10,000).
Read the interval off the B replicates.

Efron introduced this in 1979, and the textbook An Introduction to Statistical Learning (section 5.2) presents it as a way to quantify the uncertainty of almost any estimator.

Step 7: Why sampling with replacement works

Drawing without replacement n of n items just gives your original sample back in a different order, with no variation. Drawing with replacement mimics fresh draws from the population. The bootstrap pile of replicates approximates the sampling distribution centred near p̂, with about the same spread as the true one. That is why Rhea's bootstrap standard deviation (0.0356) matches the formula SE (0.0357).

Step 8: Reading the percentile interval and the out-of-bag fraction

The simplest bootstrap interval takes the 2.5th and 97.5th percentiles of the replicates. In one resample of n items, each item is missed with probability (1 − 1/n)ⁿ. For n = 100 that is 0.366, and as n grows it tends to 1/e ≈ 0.368. So each bootstrap sample contains only about 63% of the distinct items. The missing 37% are called out-of-bag, a handy free validation set (section 5).

5. Everything the video skipped

Three kinds of bootstrap interval

Let θ̂ be the statistic on the real data and θ̂* the statistics on the replicates.

Percentile: take the 2.5th and 97.5th percentiles of θ̂*. Simple, and invariant if you transform the statistic (the interval for log θ is the log of the interval for θ). It can be biased when the replicate distribution is skewed or off-centre.
Basic (also called reverse percentile or pivotal): [2θ̂ − q₉₇.₅, 2θ̂ − q₂.₅], where q are the replicate percentiles. It reflects the percentile interval around θ̂, assuming the error θ̂* − θ̂ behaves like θ̂ − θ. For a symmetric replicate pile it equals the percentile interval; for a skewed one it flips the skew.
BCa (bias-corrected and accelerated): shifts the percentile levels used, with a bias correction z₀ (how far the median of θ̂* is from θ̂) and an acceleration a (how fast the standard error changes with the parameter, usually estimated by the jackknife). It is second-order accurate and is the usual best choice for modest n and skewed statistics, but needs more computation.

scipy.stats.bootstrap offers 'percentile', 'basic' and 'BCa', with BCa as the default. Checked on Rhea's data (10,000 resamples, seed 0): percentile 0.78 to 0.92, basic 0.78 to 0.92, BCa 0.77 to 0.91. A discrete accuracy of 100 yes/no values is slightly skewed because 0.85 sits fairly near the ceiling of 1, which BCa notices.

Wald versus Wilson at small n

The Wald interval assumes a normal bell for p̂, which fails when n is small or p̂ is near 0 or 1. Three symptoms:

It can leave the legal range. With 9 of 10 right, p̂ = 0.9, SE = √(0.9 × 0.1 / 10) = 0.0949, and the interval is 0.714 to 1.086, above 100%.
It collapses at the extremes. With 10 of 10 right, p̂(1 − p̂) = 0, so SE = 0 and the interval is the single point [1, 1]. This claims perfect certainty from ten photos.
Its real coverage is lower than advertised. In a simulation (true p = 0.95, n = 20), the nominal 95% Wald interval contained the truth only about 64% of the time. The Wilson interval, calculated exactly, covers about 92.5%.

The Wilson interval, with p̂ = k/n and z = 1.96, is

centre = (p̂ + z²/(2n)) / (1 + z²/n), half-width = z √(p̂(1 − p̂)/n + z²/(4n²)) / (1 + z²/n).

For 85 of 100 it gives 0.767 to 0.907 (slightly shifted toward 0.5). For 9 of 10 it gives 0.596 to 0.982; for 10 of 10 it gives 0.722 to 1.000. Wilson never leaves [0, 1] and never claims zero uncertainty. (Wilson's 1927 interval is background knowledge, not part of the video's sources.) At n = 100 with p̂ = 0.85 the Wald and Wilson answers are close, so Rhea's story is safe; for 20 or fewer test items, prefer Wilson or an exact binomial method.

t-intervals for a mean from a small sample

For a mean of a continuous quantity (say validation scores over 5 random seeds), σ is unknown and estimated by the sample standard deviation s. The extra uncertainty makes the ratio (x̄ − μ)/(s/√n) follow a Student's t distribution with n − 1 degrees of freedom (William Gosset, 1908). The interval is

x̄ ± t* s/√n.

The t* multiplier is bigger than 1.96 for small samples: 2.776 for 4 degrees of freedom (n = 5), 2.262 for 9 (n = 10), 2.093 for 19 (n = 20), 2.045 for 29 (n = 30), 1.984 for 99. By about n = 30 it is close to 1.96, which is where the rule of thumb "n ≥ 30" comes from. Rhea's accuracy is a proportion, so her interval uses the normal curve, but a mean of scores over a few seeds should use t.

Comparing two models

Goodfellow, Bengio and Courville state a simple rule: algorithm A is better than algorithm B if the upper bound of A's 95% CI for error is below the lower bound of B's. This is safe but conservative: if the intervals overlap slightly, the models may still differ significantly. The sharper method is to look at the difference directly.

Unpaired: two separate test sets, SE_diff = √(SE_A² + SE_B²).
Paired: both models on the same photos. Count only the photos where they disagree. Pairing removes the variation caused by easy and hard photos, which affects both models equally, and usually gives a tighter interval. Example B in section 6 does this.
Bootstrap for the difference: resample the photo indices once, compute both accuracies on the same resample, and take the difference. The percentile interval of the differences is the CI for A − B. If it contains 0, you cannot claim a real improvement.

Out-of-bag (OOB) estimates

A bootstrap sample of n items leaves out each item with probability (1 − 1/n)ⁿ ≈ 1/e ≈ 0.368 (An Introduction to Statistical Learning says "roughly one-third" of observations are out-of-bag for each bagged tree). In bagging and random forests each tree is trained on its own bootstrap sample, so every data point has about 37% of the trees that never saw it. Predict that point using only those trees, and the errors average into the OOB error, which behaves like a cross-validation estimate without needing a separate hold-out set.

Formulas in simple words

Accuracy SE = √(p̂(1 − p̂)/n): "how noisy a yes/no average is".
Mean SE = s/√n: "spread of one item divided by the square root of the count".
Interval = estimate ± multiplier × SE.
Sample size for a target margin m: n ≈ (z/m)² p̂(1 − p̂). Worst case (p̂ = 0.5) needs n = (1.96/m)² × 0.25, which for ±1 point is about 9,604 items.

Limitations

A CI covers only random sampling error. If the test set is biased (wrong kind of photos, duplicates from training), the interval is a precise answer to the wrong question.
The bootstrap assumes items are independent. For time series, grouped data (many photos of the same cat) or users with several rows, resample whole groups or blocks.
The bootstrap works poorly for extremes (maximum, minimum), for very small n, and for statistics that depend on single data points.
Reporting a CI does not fix p-hacking: if you try 20 models and report the best, the interval for the best is too optimistic (see 1.40).

Background

Jerzy Neyman introduced confidence intervals in 1937. Bradley Efron's 1979 paper "Bootstrap Methods: Another Look at the Jackknife" introduced the bootstrap. The name refers to the tale of lifting yourself by your bootstraps.
Confidence intervals (frequentist) are not Bayesian credible intervals (1.38). A credible interval does say "there is a 95% probability that θ lies here, given the prior and data", which is what people often wrongly read into a confidence interval. With a flat prior and plenty of data the two numerically agree.
A confidence interval and a hypothesis test are two views of one thing: a 95% CI contains exactly those values of p that a two-sided 5% test would not reject.

Links on the roadmap

1.34 joint, marginal and independent; 1.35 expectation and covariance; 1.39 central limit theorem; 1.40 hypothesis tests and p-values; 1.38 Bayesian updating; 2.05 loss functions; 1.42 Markov chains; 1.43 Monte Carlo sampling (the bootstrap is a Monte Carlo method).

6. Worked examples with real numbers

Example A: How many test photos does Rhea need?

Setup: Rhea wants her accuracy known to within ±2 points (95% confidence), expecting p̂ ≈ 0.85.

Start from the margin formula: m = 1.96 √(p̂(1 − p̂)/n).
Solve for n: n = (1.96/m)² × p̂(1 − p̂).
Plug in m = 0.02: (1.96/0.02)² = 98² = 9,604.
Multiply by p̂(1 − p̂) = 0.85 × 0.15 = 0.1275: 9,604 × 0.1275 = 1,224.5.
Round up: n = 1,225 photos.
Check: 1.96 × √(0.1275/1,225) = 1.96 × 0.01020 = 0.0200. Good.

Result: She needs about 1,225 test photos for ±2 points. With the 100 she has, the margin is ±7.0 points. If she did not know p̂ she would use the worst case p̂ = 0.5 (0.25 instead of 0.1275), which needs 2,401 photos.

Example B: Is model A really better than model B?

Setup: Both models are tested on the same 100 photos. A gets 85 right, B gets 80 right. (Illustrative counts.) Breakdown: both right 75, only A right 10, only B right 5, both wrong 10. So 75 + 10 = 85 for A and 75 + 5 = 80 for B.

Unpaired SEs: SE_A = √(0.85 × 0.15 / 100) = 0.0357 and SE_B = √(0.80 × 0.20 / 100) = 0.0400.
Separate 95% intervals: A is 0.780 to 0.920, B is 0.722 to 0.878. They overlap a lot, so the GF5 rule cannot call A better.
Paired difference: d̂ = (10 − 5)/100 = 0.05.
Paired SE: each photo's difference is +1, −1 or 0. Var of one difference = (10 + 5)/100 − 0.05² = 0.15 − 0.0025 = 0.1475, so SE = √0.1475 / √100 = 0.0384.
Interval: 0.05 ± 1.96 × 0.0384 = 0.05 ± 0.075 → −0.025 to +0.125.
Cross-check with an exact test on the 15 disagreements: if the models were equal, A should win about half, and the chance of a split at least as lopsided as 10 to 5 (two-sided) is p = 0.30.

Result: A looks 5 points better, but the interval for the gap runs from −2.5 to +12.5 points and includes 0 (p ≈ 0.30). With 100 photos we cannot say A is really better. Gathering more photos or more disagreements is the way to settle it.

Example C: A t-interval for 5 seeds

Setup: A model is trained with 5 different random seeds and the validation accuracies (%) are 82, 88, 79, 91, 85. (Illustrative numbers.)

Mean: x̄ = (82 + 88 + 79 + 91 + 85)/5 = 85.0.
Sample standard deviation (÷ n − 1): s = 4.743.
Standard error: s/√5 = 4.743/2.236 = 2.121.
t multiplier for 4 degrees of freedom: t* = 2.776.
Margin: 2.776 × 2.121 = 5.89.
Interval: 85.0 ± 5.89 → 79.1 to 90.9.
Compare with the wrong normal multiplier: 85.0 ± 1.96 × 2.121 gives 80.8 to 89.2, which is too narrow.

Result: The mean accuracy is 85% with 95% CI 79.1% to 90.9%. Using 1.96 instead of 2.776 would understate the uncertainty by about 30%.

7. Comparison tables

Interval methods

Method	Needs a formula?	Best when	Weak when
Wald (normal)	Yes	n large, p̂ in the middle	small n, p̂ near 0 or 1
Wilson	Yes	proportions, any n	rarely a problem
Student's t	Yes	mean of a few values	skewed data, tiny n
Bootstrap percentile	No	any statistic, moderate n	very small n, skew
Bootstrap BCa	No	skewed, biased statistics	needs more compute

Width of Rhea's interval as n grows

Test items n	SE	Margin (95%)	Interval for p̂ = 0.85
100	0.0357	±7.0 pts	0.780 to 0.920
400	0.0179	±3.5 pts	0.815 to 0.885
1,000	0.0113	±2.2 pts	0.828 to 0.872
10,000	0.0036	±0.7 pts	0.843 to 0.857

Idea	What it says	Randomness lives in
Confidence interval	method covers the truth 95% of the time	the interval
Bayesian credible interval	95% probability the parameter is inside	the parameter
Standard deviation	spread of individual items	the data
Standard error	spread of the estimate	the estimate

8. Common mistakes and myths

Myth: "There is a 95% chance the true accuracy is in 0.78 to 0.92." The truth is fixed; the 95% belongs to the procedure. This is the first of the three fallacies described by Morey et al. and quoted in D2L.
Myth: "A narrow interval means the model is accurate." It means the estimate is precise, which is not the same as being good. A model with 51% accuracy on 100,000 items has a very narrow interval around a poor number.
Myth: "A wide interval means we know nothing." It tells you how much you know; here the model is clearly better than 70% but not clearly better than 90%.
Mistake: mixing up SD and SE. The SD describes individual items; the SE describes the estimate and shrinks with √n.
Mistake: "Two overlapping intervals mean no difference." Overlap is a conservative check. Use the interval of the difference, ideally paired.
Mistake: "Non-overlapping intervals prove a big difference." They show a real difference, not necessarily a large or useful one.
Mistake: using Wald for tiny n or p̂ of 0 or 1. Switch to Wilson or an exact interval.
Mistake: resampling the wrong unit. If one cat appears in 5 photos, resample cats, not photos.
Mistake: too few bootstrap replicates. 100 is too few for tail percentiles; use at least 1,000, and 10,000 for 95% tails.
Mistake: forgetting the test set must be unseen. If any test photo leaked into training, the interval is accurate for the wrong thing.
Myth: "The bootstrap creates new data." It creates no new information; it only reuses what you have to measure its variability.

9. Real-life and AI uses

Model evaluation: reporting accuracy, F1 or AUC with a CI so that readers know whether 85% versus 86% is real.
A/B tests: comparing click-through rate of two versions of a page with a CI on the difference.
Benchmarks for language models: a leaderboard gap of 0.5 points on a 500-question benchmark is often smaller than its margin of error (at n = 500 and p̂ ≈ 0.7, margin ≈ ±4 points).
Random forests and bagging: each tree is trained on a bootstrap sample, and the out-of-bag error acts as free validation.
Reproducibility: running training with several seeds and reporting mean ± t-interval, not the best seed.
Polls and medicine: "±3 percentage points" in a poll is a 95% margin of error; clinical trials report CIs for effect sizes.
Uncertainty of any odd statistic: median latency, 99th percentile response time, a Gini index, a ratio of two means, where there is no neat formula but the bootstrap works.

10. Self-check questions (with answers)

What is the standard error of Rhea's accuracy, and how is it computed? SE = √(p̂(1 − p̂)/n) = √(0.85 × 0.15 / 100) ≈ 0.0357.
What is the 95% CI for 85 of 100 correct? 0.85 ± 1.96 × 0.0357 = 0.85 ± 0.070, so 0.780 to 0.920.
What does "95% confidence" mean? If we repeated the experiment many times and built an interval each time, about 95% of the intervals would contain the true accuracy. It is not the probability that one computed interval is right.
If Rhea wants to halve her margin of error, how many more photos does she need? Four times as many, since the width is proportional to 1/√n (400 photos give ±3.5 points).
Why does the bootstrap sample with replacement? Without replacement the resample would just be the original data reordered, with no variation. Sampling with replacement mimics drawing fresh data from the population.
About what fraction of the items is out-of-bag in one bootstrap sample? (1 − 1/n)ⁿ → 1/e ≈ 0.368, about 37% (0.366 for n = 100).
Name three bootstrap interval types and one difference between them. Percentile, basic and BCa. Percentile takes the replicate percentiles directly; basic reflects them around θ̂; BCa adjusts the percentile levels for bias and skew.
Why is the Wald interval a bad choice for 10 of 10 correct? p̂(1 − p̂) = 0 so SE = 0 and the interval collapses to [1, 1], claiming certainty. Wilson gives about 0.72 to 1.00.
Why use t* = 2.776 rather than 1.96 for 5 seeds? σ is estimated from only 5 points (4 degrees of freedom), and the t distribution has heavier tails; using 1.96 would give an interval that is too narrow.
Two models have 95% CIs of 0.78 to 0.92 and 0.72 to 0.88. Can we say the first is better? No. The intervals overlap, and the right check is the CI of the difference (ideally paired). In Example B that interval is −0.025 to +0.125, which includes 0.

11. Key takeaways, go deeper, next

Key takeaways

A measured score is one draw from a cloud of possible scores; the standard error measures the cloud's width.
For accuracy, SE = √(p̂(1 − p̂)/n) and the 95% CI is p̂ ± 1.96 SE.
The "95%" describes the method, not the single interval.
Width shrinks like 1/√n: four times the data, half the error.
The bootstrap builds an interval by resampling your data with replacement, needing no formula; percentile, basic and BCa are the common recipes.
At small n or near 0 and 1 use Wilson (proportions) or t (means of few values).
To compare models, build an interval for the difference, preferably paired.
About 37% of items are out-of-bag in each bootstrap sample, a free validation set for bagging.

Go deeper

Read An Introduction to Statistical Learning section 5.2 for the bootstrap and try scipy.stats.bootstrap with the three methods on your own model's per-item results.
Read Efron's 1979 paper (the first sections are readable) for the original idea.
Simulate the coverage of Wald and Wilson intervals at n = 20, p = 0.95 to see 64% versus 92% for yourself.

Next on the roadmap: 1.42 Markov chains.

12. Sources and further reading

Zhang, Lipton, Li, Smola, Dive into Deep Learning (2023), 22.10 Statistics: estimators, hypothesis tests, confidence intervals
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 5 Machine Learning Basics: 5.4 Estimators, bias and variance, standard error
James, Witten, Hastie, Tibshirani, Taylor, An Introduction to Statistical Learning with Applications in Python (2023), 5.2 The Bootstrap and 8.2 Bagging
Efron, B., Bootstrap Methods: Another Look at the Jackknife, Annals of Statistics 7(1) (1979), full paper PDF
SciPy documentation, scipy.stats.bootstrap (percentile, basic and BCa methods), reference page
Zhang, Lipton, Li, Smola, Dive into Deep Learning (2023), 22.8 Distributions: binomial tends to the Gaussian

Accuracy note: The cat/dog classifier, the 85 of 100 score, the 5-seed validation scores, and the paired 75/10/5/10 model comparison are illustrative data made up for teaching; all arithmetic on them (SE, intervals, Wilson, t, bootstrap, out-of-bag 0.366) was recomputed in Python with numpy and scipy. The facts about the definition of a confidence interval, Neyman 1937, the 1.96 recipe, the model-comparison rule, the bootstrap, and the out-of-bag fraction come from the sources above. The Wilson interval, the Wald coverage simulation (about 64% versus 92.5%), BCa details, Gosset's t, the paired-difference formula and the CI-test duality are background knowledge, not taken from those sources.

Confidence intervals and bootstrapping: study notes