1.40 Hypothesis testing and p-values: study notes

Learn AI by Story

1. The idea in one sentence

Hypothesis testing asks: "if nothing special were going on, how surprising would my data be?", and the p-value is the number that answers it. Plain English: it is a way to separate a lucky streak from a real effect.

2. The story from the video

Zara builds a model that predicts whether a stock will go up or down tomorrow. She tests it on 100 days and it is right on 60 of them. Is that skill, or luck? A coin that guesses at random is right half the time, so 50 of 100 would be typical, but 60 is not impossible either.

Zara sets up a courtroom. The "innocent until proven guilty" default is the null hypothesis H₀: the model is just guessing, p = 0.5. The claim she hopes to support is the alternative H₁: p > 0.5. If H₀ is true, the number of correct days X follows a Binomial(100, 0.5) distribution. She asks: how likely is a result at least as good as 60 if H₀ is true? The exact answer is P(X ≥ 60) = 0.0284, about 2.8%. That is the p-value. Since 0.0284 is below the usual significance level α = 0.05, she rejects H₀ and says the result is statistically significant.

The video then adds three warnings. First, the p-value is not the probability that the model is useless. Second, the two-sided version of the same test gives p = 0.0569, which would not pass 0.05, so the way you set up the question matters. Third, run enough tests and something will look significant by chance alone.

3. Key definitions (glossary)

Null hypothesis H₀: the default "nothing is going on" claim, here p = 0.5. Plain English: the boring explanation we try to knock down.
Alternative hypothesis H₁: what we accept if the data make H₀ hard to believe, here p > 0.5 (one-sided) or p ≠ 0.5 (two-sided). Plain English: the interesting claim.
Test statistic: a number computed from the data, such as the count X or a z-score. Plain English: a single score that summarises how far the data are from H₀.
Sampling distribution under H₀: the distribution of the test statistic if H₀ were true. Plain English: what "just luck" looks like.
p-value: the probability, assuming H₀ is true, of getting a test statistic at least as extreme as the one observed [ISL ch. 13]. Written P(data this extreme or more, given H₀). Plain English: how embarrassed H₀ should feel.
Significance level α: the cut-off chosen before the test; we reject H₀ if p ≤ α. Commonly α = 5% [D2L-STAT]. Plain English: the false-alarm rate we are willing to live with.
Type I error: rejecting H₀ when it is true (a false positive). Its probability is at most α. Plain English: crying wolf.
Type II error: failing to reject H₀ when it is false (a false negative). Its probability is β. Plain English: missing a real effect.
Power: 1 − β, the probability of rejecting H₀ when H₁ is true; commonly aimed at 80% [D2L-STAT]. Plain English: the chance of catching a real effect.
Rejection region: the set of results for which we reject H₀. Plain English: the "too surprising" zone.
One-sided and two-sided tests: one-sided looks for a difference in one direction only; two-sided counts both directions. Plain English: "better" versus "different".
Multiple testing: running many tests at once, which inflates the chance of at least one false positive. Plain English: more lottery tickets, more fake winners.
Effect size: how large the difference is (for Zara, 60% versus 50%). Plain English: how much it matters, as opposed to whether it is detectable.

4. Detailed explanation, from basics to the full picture

Step 1: Pick the null and the alternative

State H₀ as a precise, checkable claim (p = 0.5) and H₁ as what you would conclude otherwise. Decide one-sided or two-sided before seeing the data. Zara cares only about "better than guessing", so H₁: p > 0.5.

Step 2: Choose α first

Write down α (for example 0.05) before looking at the results. This is the false-alarm rate you accept. Choosing it after seeing the data lets you pick whichever threshold gives the answer you want.

Step 3: Compute the test statistic

For Zara the natural statistic is X, the number of correct days: X = 60.

Step 4: Work out what H₀ predicts

Under H₀, each day is a fair coin flip, so X ~ Binomial(100, 0.5). The mean is 50 and the standard deviation is √(100 × 0.5 × 0.5) = 5. The value 60 is 2 standard deviations above the mean, matching the central limit theorem picture of video 1.39.

Step 5: Compute the p-value

The one-sided p-value is P(X ≥ 60) = 0.0284 (exact, from scipy's binomtest [SCIPY-B]). Note the "or more extreme" part: we add up the probability of 60, 61, ..., 100, not just of 60. Getting exactly 60 has probability only about 0.0108, but that is not the right quantity, because every single outcome is individually improbable.

Step 6: Decide

If p ≤ α, reject H₀; otherwise, fail to reject. Here 0.0284 ≤ 0.05, so reject. Never say "accept H₀": failing to reject means the evidence was not strong enough, not that H₀ is proven.

Step 7: The rejection region view

Equivalent to Step 5: find the smallest X such that P(X ≥ x) ≤ 0.05. We have P(X ≥ 59) = 0.0443 and P(X ≥ 58) = 0.0666, so the rejection region is X ≥ 59. Any result of 59 or more correct days leads to rejection at α = 0.05. The true false-alarm rate is 0.0443, slightly below 0.05 because the binomial is discrete.

Step 8: Think about power

Suppose the model really is right 60% of the time. Then the probability of landing in the rejection region is P(X ≥ 59 given p = 0.6) = 0.6225. So the power is about 62% and β ≈ 38%: even a genuine 60% model fails this test 38% of the time on a 100-day sample. A common target is power of 80% [D2L-STAT], which needs more data (see Example C).

Step 9: Report more than "significant"

Report the effect size (60%), a confidence interval (video 1.41) and the p-value, not just the verdict.

5. Everything the video skipped

Formulas in simple words

Exact binomial tail: P(X ≥ k) = ∑ over j from k to n of C(n, j) pʲ (1 − p)ⁿ⁻ʲ. Add the probability of every outcome at least as extreme.
Normal approximation (z-test): z = (X − np₀)/√(np₀(1 − p₀)). Subtract the expected count and divide by the standard deviation. Then look up z on the normal curve (CLT).
Two-sided p-value: add the tail on the other side too. For a symmetric null like p₀ = 0.5 it is exactly double the one-sided value: 2 × 0.0284 = 0.0569.
Type I rate: P(reject given H₀) ≤ α. Power: P(reject given H₁) = 1 − β.
Family-wise error for m independent tests at level α: 1 − (1 − α)ᵐ.
Bonferroni: reject H₀ⱼ only if pⱼ < α/m [ISL 13.3]. Holm's method is more powerful and equally safe about the family-wise error rate.

Variants

One-sample binomial test (Zara's), z-test and t-test for means (t when σ is unknown and n is small), two-sample tests (compare two groups, for example two models on the same test set), paired tests, chi-squared test for counts and categories.
Permutation tests: shuffle labels to build the null distribution directly, with no formula.
Bayesian alternatives: compute the probability of each hypothesis from a prior (video 1.38), which answers the question people often wish the p-value answered.

Limitations

Binomial discreteness: with n = 100 you cannot hit α = 0.05 exactly (0.0443 is the nearest below).
A tiny p-value does not mean a big effect. Big n makes tiny effects significant (Example B).
A big p-value does not prove H₀; it may just mean low power.
All the logic assumes the data were collected as the model says (independent draws, fixed n, the one test planned in advance).

Background

John Arbuthnot used 1700s birth records to argue that births were not 50:50, an early significance-style argument. Karl Pearson developed the p-value and the chi-squared test, William Gosset ("Student") the t-test, and Ronald Fisher the null hypothesis and significance testing [D2L-STAT]. Jerzy Neyman added confidence intervals in 1937 [D2L-STAT]; the Type I, Type II and power framework comes from the Neyman-Pearson tradition (background).
In 2016 the American Statistical Association published six principles on p-values [ASA]: p-values can indicate how incompatible data are with a model; they do NOT measure the probability the hypothesis is true or that the data came from chance alone; decisions should not rest only on crossing a threshold; full reporting and transparency are needed; a p-value does not measure effect size or importance; and by itself it is not a good measure of evidence.
Stockbroker trap [ISL 13.2]: a broker emails 1,024 = 2¹⁰ clients. Half are told "up", half "down", and only those with correct calls stay on the list. After 10 rounds one client has seen 10 correct calls in a row, by pure luck. If you only see the winner, p = (1/2)¹⁰ ≈ 0.001 looks impressive, but you ignored the 1,023 who were not shown.

Links on the roadmap

1.39 The central limit theorem: why test statistics are approximately normal.
1.41 Confidence intervals and bootstrapping: the companion tool that shows effect size and uncertainty.
1.38 Bayesian updating: the alternative way to weigh hypotheses.
1.43 Monte Carlo sampling: simulate the null to get p-values for any statistic.

6. Worked examples with real numbers

Example A: Zara's exact test

Setup: H₀: p = 0.5, H₁: p > 0.5, n = 100, observed X = 60, α = 0.05.

Under H₀, X ~ Binomial(100, 0.5), with mean 50 and standard deviation 5.
One-sided p-value: P(X ≥ 60) = 0.0284.
Compare: 0.0284 ≤ 0.05, so reject H₀.
Two-sided p-value: 2 × 0.0284 = 0.0569.
Compare: 0.0569 > 0.05, so a two-sided test would not reject.
Rejection region at α = 0.05: P(X ≥ 59) = 0.0443 ≤ 0.05 but P(X ≥ 58) = 0.0666 > 0.05, so reject when X ≥ 59.
Power if the true accuracy is 0.6: P(X ≥ 59 given p = 0.6) = 0.6225.

Result: significant at 5% one-sided (p = 0.0284), not significant two-sided (p = 0.0569), power about 62%. The conclusion depends on the question you committed to in advance, and 60 of 100 is borderline evidence.

Example B: Big sample, small effect

Setup: a second model is right on 5,100 of 10,000 days (51%). Test H₀: p = 0.5 against H₁: p > 0.5.

Under H₀: mean = 5,000, standard deviation = √(10,000 × 0.25) = 50.
z-score: z = (5,100 − 5,000)/50 = 2.0.
Normal-approximation one-sided p-value: 0.0228.
Exact binomial one-sided p-value: 0.0233.
Two-sided exact p-value: 0.0466.

Result: p = 0.0228 (normal) or 0.0233 (exact) is significant at 5%, yet the model is right only 51% of the time, one percentage point above guessing. Significant does not mean useful. The same 1% edge on 100 days would give p ≈ 0.46 (not significant); the statistical evidence comes from n, not from the size of the effect.

Example C: How many days for 80% power?

Setup: Zara wants to detect a true accuracy of 0.6 with power near 80% at α = 0.05 one-sided. We search for n, using exact binomial calculations.

n = 100: rejection region X ≥ 59 (false-alarm rate 0.0443), power 0.6225.
n = 200: rejection region X ≥ 113 (false-alarm rate 0.0384), power 0.8603.
n = 400: rejection region X ≥ 217 (false-alarm rate 0.0494), power 0.9915.
n = 160: rejection region X ≥ 91, power 0.813. Power zigzags slightly with n because the binomial is discrete (n = 150 gives 0.774).

Result: about 160 days are needed to catch a true 60% model 80% of the time; 100 days is underpowered (62%).

Example D: Twenty tests

Setup: a team compares 20 independent feature variations to a baseline, each tested at α = 0.05, when in truth none of them does anything.

Probability that one test gives no false alarm: 0.95.
Probability that all 20 give none: 0.95²⁰ = 0.358.
Probability of at least one false alarm: 1 − 0.358 = 0.642.
Expected number of false alarms: 20 × 0.05 = 1.
Bonferroni threshold: 0.05/20 = 0.0025 per test [ISL 13.3].
Family-wise error with Bonferroni: 1 − (1 − 0.0025)²⁰ = 0.0488, just under 0.05.

Result: without correction there is a 64% chance of at least one "significant discovery" by pure luck; with Bonferroni, the chance drops to about 4.9%.

7. Comparison tables

The four outcomes of a test

Situation	Reject H₀	Fail to reject H₀
H₀ true	Type I error (probability α)	Correct
H₀ false	Correct (power 1 − β)	Type II error (probability β)

Zara's tail probabilities

Result	P(X ≥ k) under H₀	Meaning
k = 58	0.0666	Not significant at 5%
k = 59	0.0443	Smallest significant result
k = 60	0.0284	Observed, one-sided p
two-sided	0.0569	Not significant at 5%

Misreadings of the p-value

Wrong reading	Correct reading	Why it fails
P(H₀ is true) = 0.028	P(data this extreme given H₀) = 0.028	Mixes up the direction of conditioning
97.2% sure the model works	Data are unusual if guessing	Ignores prior plausibility
Large effect	Possibly tiny effect	p depends on n
p > 0.05 proves H₀	No clear evidence either way	Could be low power

8. Common mistakes and myths

Myth: p = 0.03 means a 3% chance that H₀ is true. It is the chance of data this extreme if H₀ were true [ASA]. To get P(H₀ given data) you need a prior and Bayes' rule (video 1.38).
Myth: p-value is the probability the result is due to chance. The ASA explicitly warns against this reading [ASA].
Mistake: treating 0.05 as a magic line. 0.049 and 0.051 are practically the same evidence [ASA].
Mistake: p-hacking. Trying many analyses, subgroups or thresholds and reporting only the one with p < 0.05 inflates false positives (Example D).
Mistake: switching to a one-sided test after seeing the data. Zara's one-sided p = 0.0284 passes, her two-sided p = 0.0569 does not; choose before looking.
Mistake: confusing significant with important. Example B is significant but nearly useless.
Mistake: "not significant" means "no effect". A low-power test (62% in Example A, if the effect is real) misses real effects often.
Mistake: ignoring the stockbroker effect. If you only see the winners of many trials, p-values are meaningless [ISL 13.2].
Mistake: reusing the test set repeatedly. Tuning a model on the same test data many times is multiple testing in disguise.

9. Real-life and AI uses

Comparing models. Is model A really better than B, or did it get lucky on this test set? A paired test or a confidence interval for the difference gives the answer; [GF5] section 5.4.4 suggests comparing 95% intervals.
A/B tests. Is the new button's conversion rate different from the old one? Tests on proportions, built on the CLT.
Feature selection. Test whether each feature has a relationship with the target; with thousands of features, correct for multiple testing (Bonferroni, Holm) [ISL ch. 13].
Medicine and science. Clinical trials test the drug against placebo with a pre-set α and power, often 80% [D2L-STAT].
Fairness and drift checks. Test whether error rates differ between groups, or whether today's data distribution differs from training data.
Benchmark claims. A 0.5-point leaderboard gain on a few thousand examples is often inside the noise; a test shows whether it is.
Spam and anomaly alerts. Each alert is a hypothesis test; many tests per day means many false alarms unless thresholds are corrected.

10. Self-check questions (with answers)

State H₀ and H₁ in Zara's story. H₀: p = 0.5 (the model is guessing); H₁: p > 0.5 (it is better than guessing).
What is the p-value, in words? The probability, assuming H₀ is true, of a result at least as extreme as the one observed.
Zara's one-sided p-value is 0.0284. What is the two-sided value and what changes? It is 2 × 0.0284 = 0.0569, which is above 0.05, so the conclusion flips to "not significant".
What is the rejection region at α = 0.05 with n = 100? X ≥ 59, because P(X ≥ 59) = 0.0443 ≤ 0.05 while P(X ≥ 58) = 0.0666 > 0.05.
Does p = 0.0284 mean there is a 2.84% chance that the model is useless? No. It is the chance of data this extreme if the model were guessing; it is not the probability of H₀.
What are Type I and Type II errors? Type I: rejecting a true H₀ (false positive, probability at most α). Type II: failing to reject a false H₀ (false negative, probability β).
What is the power of the test when the true accuracy is 0.6? About 0.6225, so β ≈ 0.38.
5,100 correct out of 10,000 gives z = 2.0. Is that a strong model? It is statistically significant (p ≈ 0.023) but the model is only 1 point above guessing, so the effect is small.
With 20 independent tests at α = 0.05 and no real effects, what is the chance of at least one false alarm, and how does Bonferroni help? 1 − 0.95²⁰ ≈ 0.642. Bonferroni tests each at 0.05/20 = 0.0025, bringing the overall chance back to about 5%.
Name two of the ASA's warnings about p-values. For example: a p-value is not the probability that the hypothesis is true, and a conclusion should not depend only on whether p crosses a threshold such as 0.05; also, p does not measure effect size or importance.

11. Key takeaways, go deeper, next

Key takeaways

Hypothesis testing: assume H₀, find how surprising the data are, reject if p ≤ α.
p-value = P(data this extreme given H₀), not P(H₀ given data).
Type I error is capped by α; power (1 − β) depends on effect size and n; Zara's 100-day test has only about 62% power for a true 60% model.
Direction (one-sided versus two-sided) must be chosen before the data.
Many tests mean many false alarms: use Bonferroni or Holm.
Always report the effect size and an interval, not just "significant".

Go deeper

Run scipy.stats.binomtest(60, 100, 0.5, alternative="greater") and vary k and n [SCIPY-B].
Simulate 100,000 experiments under H₀ and check that about 5% give p ≤ 0.05.
Read the ASA's six principles [ASA] and ISL chapter 13 on multiple testing.

Next on the roadmap: 1.41 Confidence intervals and bootstrapping.

12. Sources and further reading

James, Witten, Hastie, Tibshirani, Taylor, An Introduction to Statistical Learning with Applications in Python (2023), Chapter 13, Multiple Testing
Zhang, Lipton, Li, Smola, Dive into Deep Learning (online book), 22.10 Statistics
ScienceDaily (2016), American Statistical Association releases statement on statistical significance and p-values, ASA statement summary
SciPy documentation, scipy.stats.binomtest, binomtest reference
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 5, Machine Learning Basics, section 5.4.4 Standard error
Zhang, Lipton, Li, Smola, Dive into Deep Learning (online book), 22.8 Distributions

Accuracy note: Zara's story (60 of 100 days) and the 5,100 of 10,000 case are illustrative. All probabilities (0.0284, 0.0569, 0.0443, 0.0666, 0.6225, 0.0228, 0.0233, 0.0466, 0.642, 0.0025 and the power values for n = 200 and 400) were computed and checked in Python with scipy. The facts about the ASA's principles, the history of tests, the 1,024-client stockbroker story and Bonferroni come from the cited sources; the Neyman-Pearson attribution, the permutation-test and Bayesian alternatives, and the remark about repeated test-set reuse are background knowledge.

Hypothesis testing and p-values: study notes