1.38 Bayesian updating: prior to posterior: study notes

Learn AI by Story

1. The idea in one sentence

Bayesian updating starts with a prior belief, multiplies it by how well each possibility explains the new evidence (the likelihood), and renormalises to get a posterior belief: P(H | D) ∝ P(D | H) × P(H).

2. The story from the video

Arun takes a medical screening test and it comes back positive. He panics. His doctor draws 10,000 people on a board. The disease affects 1% of people, so 100 are sick and 9,900 are healthy. The test catches 90% of the sick (90 true positives) and wrongly flags 5% of the healthy (495 false positives). Among everyone who tests positive, 90 + 495 = 585 people, only 90 are really sick, so P(sick | +) = 90/585 = 0.1538, about 15%. The doctor orders a second, independent test. It is also positive. Now the old 15% becomes the new prior, and the posterior rises to about 77%.

In the second half, Asha is opening a cafe and wants to know what fraction of customers will like her new recipe. She starts with a gentle prior, Beta(2, 2), centred on 0.5. Her first 3 tasters all like it. A pure frequency count says 3 of 3 = 100%, which is silly; the Bayesian posterior Beta(5, 2) has mean 0.714 and peak 0.8. Then 20 more tasters arrive: 14 like it, 6 do not. The posterior becomes Beta(19, 8), with mean 0.704 and peak 0.72. The prior was a seat belt early on and faded as data piled up.

3. Key definitions (glossary)

Prior P(H): your belief about a hypothesis or parameter before seeing the new data. Plain English: where you start.
Likelihood P(D | H): how probable the observed data are if H is true. Plain English: how well H predicts what happened.
Evidence P(D): the overall probability of the data, averaged over all hypotheses: P(D) = Σ P(D | H) P(H). Plain English: the normaliser that makes the posterior add up to 1.
Posterior P(H | D): belief after seeing the data. Plain English: where you end up.
Bayes' rule: P(H | D) = P(D | H) P(H) ÷ P(D). Plain English: flip a conditional probability.
Prevalence (base rate): how common the condition is before testing. Plain English: how many people are sick to begin with.
Sensitivity: P(+ | sick). Plain English: how often the test catches the sick.
False positive rate: P(+ | healthy). Plain English: how often the test cries wolf.
Beta(a, b) distribution: a distribution on [0, 1] with density proportional to p^(a−1) (1 − p)^(b−1). Plain English: a flexible bell-ish shape for an unknown probability.
Conjugate prior: a prior for which the posterior is in the same family. Plain English: the update is just adding counts.
Posterior mean: the average of the posterior; for Beta(a, b) it is a ÷ (a + b). Plain English: the balanced guess.
MAP (maximum a posteriori): the mode of the posterior, θ_MAP = argmax of ln P(D | θ) + ln P(θ). Plain English: the single most plausible value after including the prior.
Sequential updating: using yesterday's posterior as today's prior. Plain English: each new fact nudges the belief.
Odds: P ÷ (1 − P). Plain English: "1 to 5" style numbers.
Likelihood ratio (Bayes factor): P(D | H₁) ÷ P(D | H₂). Plain English: how many times better H₁ predicts the data.

4. Detailed explanation, from basics to the full picture

Step 1: Start from conditional probability

By definition, P(H and D) = P(D | H) P(H) = P(H | D) P(D). Divide both sides by P(D) and you get Bayes' rule: P(H | D) = P(D | H) P(H) ÷ P(D). Nothing mystical: it is the product rule, read in two directions.

Step 2: The natural-frequency picture

Probabilities are easier as counts. Imagine 10,000 people. With 1% prevalence: 100 sick and 9,900 healthy. Sensitivity 90% gives 90 true positives. A 5% false positive rate on 9,900 healthy people gives 495 false positives. All positives together: 585. The fraction who are sick is 90 ÷ 585 = 0.1538. Even though the test is "90% accurate", a positive result means only about 15% chance of disease, because healthy people vastly outnumber sick ones. This is the base-rate effect.

Step 3: Why a second test changes so much

After the first positive, the prior for Arun is no longer 1% but 0.1538. Apply the same test again (assuming the two results are independent given his true state): numerator 0.9 × 0.1538 = 0.1385; denominator 0.1385 + 0.05 × 0.8462 = 0.1385 + 0.0423 = 0.1808. Posterior = 0.1385 ÷ 0.1808 ≈ 0.766, about 77%. Yesterday's posterior is today's prior.

Step 4: Updating a continuous parameter

For an unknown probability p (like the cafe's like-rate), the prior is a curve over [0, 1]. Multiply the curve by the likelihood of the data and renormalise. For m likes and l dislikes the likelihood is proportional to p^m (1 − p)^l. If the prior is Beta(a, b), proportional to p^(a−1) (1 − p)^(b−1), then the product is proportional to p^(a+m−1) (1 − p)^(b+l−1), which is Beta(a + m, b + l). Updating is just counting: add likes to a, dislikes to b (Bishop, 2.1.1).

Step 5: Reading the Beta

Beta(a, b) has mean a ÷ (a + b) and, for a, b > 1, mode (a − 1) ÷ (a + b − 2). Think of a and b as pseudo-counts: Beta(2, 2) is "as if I had already seen 2 likes and 2 dislikes, weakly". Larger a + b means a more confident, narrower belief.

Step 6: Prior fades, data takes over

The posterior mean lies between the prior mean and the MLE (Bishop, 2.1.1). With few data the prior has a big say; with many data the likelihood dominates and the posterior approaches the MLE. In Asha's case after 3 of 3 the MLE is 1.00 but the posterior mean is 0.714; after 20 more tasters the posterior mean is 0.704, now close to the observed 17 of 23 = 0.739 and far from the starting 0.5.

Step 7: MAP and regularisation

Maximum a posteriori estimation maximises ln P(D | θ) + ln P(θ): the log-likelihood plus a log-prior penalty. If the prior on weights w is Gaussian N(w; 0, (1/λ) I), the penalty is proportional to λ ‖w‖², which is exactly weight decay (Goodfellow et al., 5.6.1). The same source notes that MAP lowers variance at the cost of some bias.

Step 8: Frequentist versus Bayesian wording

A frequentist treats θ as a fixed unknown number and the data as random. A Bayesian treats θ as uncertain, described by a distribution, and the data as fixed once observed (Goodfellow et al., 5.6). Both use the likelihood; the Bayesian adds the prior.

5. Everything the video skipped

Formulas in simple words

Bayes' rule: posterior ∝ likelihood × prior. "New belief is old belief times how well it predicted the data."
Odds form: posterior odds = prior odds × likelihood ratio. For Arun: prior odds 1/99, likelihood ratio 0.9 ÷ 0.05 = 18, so posterior odds = 18/99 = 0.1818, which is probability 0.1818 ÷ 1.1818 = 0.1538.
Beta update: Beta(a, b) plus m successes and l failures gives Beta(a + m, b + l).
Posterior mean: (a + m) ÷ (a + b + m + l). It is a weighted average of the prior mean and the data fraction.
MAP: θ_MAP = argmax of [ln P(D | θ) + ln P(θ)].

Choosing a prior

Flat prior, such as Beta(1, 1): lets the data speak; the posterior mode equals the MLE.
Weakly informative prior, such as Beta(2, 2): a gentle pull toward 0.5 that avoids extreme answers.
Informative prior: encodes real knowledge, such as last year's data. It must be justified, because it can dominate small data sets.
A prior that gives probability 0 to a possibility can never be updated away from that decision, however much data arrives.

Order does not matter

If observations are independent, updating on them one by one in any order, or all at once, gives the same posterior. Asha's 3 likes then 20 more gives the same Beta(19, 8) as processing all 23 at once from Beta(2, 2): 2 + 17 = 19 and 2 + 6 = 8.

Credible intervals

A Bayesian summarises uncertainty with a credible interval: a range holding, say, 95% of the posterior probability. For Beta(19, 8) it is about 0.52 to 0.86. Unlike a confidence interval (video 1.41), it directly says "given the prior and data, there is a 95% chance p is in this range".

Limitations

Results depend on the prior, which can be subjective; check how sensitive the answer is.
Computing the evidence P(D) can be hard for complex models, needing approximations such as sampling (video 1.43 Monte Carlo sampling).
Independence assumptions (two tests being independent) must hold, or the second update is too confident.
Base rates must be right: using the wrong prevalence gives the wrong posterior.

Background

Thomas Bayes' rule was published after his death in 1763 and developed by Laplace (background, not in the listed sources). Uses beyond the video: Bayesian optimisation for tuning hyperparameters, and Bayesian A/B testing, which gives "probability that variant B is better" (both background topics).

Links on the roadmap

1.34 joint, marginal and independent events (the product rule), 1.37 maximum likelihood (the likelihood part), 1.35 expectation (posterior mean), 2.05 loss functions (weight decay as a penalty), 1.43 Monte Carlo sampling (computing hard posteriors), 1.39 the central limit theorem (next).

6. Worked examples with real numbers

Example A: Arun's positive test, twice

Setup (illustrative test): prevalence 1%, sensitivity 90%, false positive rate 5%.

Take 10,000 people: 100 sick, 9,900 healthy.
True positives: 100 × 0.90 = 90.
False positives: 9,900 × 0.05 = 495.
All positives: 90 + 495 = 585.
First posterior: 90 ÷ 585 = 0.1538.
Second test, prior 0.1538: numerator 0.90 × 0.1538 = 0.1385; denominator 0.1385 + 0.05 × 0.8462 = 0.1808.
Second posterior: 0.1385 ÷ 0.1808 = 0.766.

Result: about 15% after one positive test, about 77% after two independent positives.

Example B: Asha's cafe recipe with a Beta prior

Setup: prior Beta(2, 2) for the like-rate p; data arrive in two batches.

Prior: mean 2 ÷ 4 = 0.5; peak density 1.5 at p = 0.5.
First batch: 3 likes, 0 dislikes gives Beta(2 + 3, 2 + 0) = Beta(5, 2).
Posterior mean: 5 ÷ 7 = 0.714. Mode: (5 − 1) ÷ (5 + 2 − 2) = 4 ÷ 5 = 0.8. Peak density 2.458.
For comparison, the MLE is 3 ÷ 3 = 1.00, an extreme.
Second batch: 14 likes, 6 dislikes gives Beta(5 + 14, 2 + 6) = Beta(19, 8).
Posterior mean: 19 ÷ 27 = 0.704. Mode: 18 ÷ 25 = 0.72.

Result: the prior tamed the 3-of-3 over-fit (0.714 instead of 1.00), and after 23 tasters the belief settles near 0.70 to 0.72.

Example C: Same prior, a 10-trial data set

Setup (illustrative): another cafe uses Beta(2, 2) and sees 8 likes in 10 tasters.

Posterior: Beta(2 + 8, 2 + 2) = Beta(10, 4).
Posterior mean: 10 ÷ 14 = 0.714.
MLE: 8 ÷ 10 = 0.800.
Prior mean: 0.5.
Check that the posterior mean lies between: 0.5 < 0.714 < 0.8.
Weighted view: (4 × 0.5 + 10 × 0.8) ÷ 14 = (2 + 8) ÷ 14 = 0.714.

Result: the posterior mean 0.714 is a compromise between the prior (0.5) and the data (0.8), weighted by the pseudo-counts 4 and 10.

Example D: What if the disease is more common?

Setup (illustrative): same test, but prevalence 10%.

Per 10,000: 1,000 sick, 9,000 healthy.
True positives: 900. False positives: 9,000 × 0.05 = 450.
Posterior: 900 ÷ 1,350 = 0.667.

Result: a higher base rate lifts the same positive result from 15% to 67%; the prior matters as much as the test.

7. Comparison tables

Prior, likelihood, posterior

Piece	Symbol	Question answered	Arun example
Prior	P(H)	before data?	1% sick
Likelihood	P(D given H)	how well does H predict?	90% versus 5%
Posterior	P(H given D)	after data?	15%

MLE, MAP and full Bayes

Method	Returns	Uses prior?	Example answer
MLE	one value	no	1.00
MAP	one value	yes	0.80
Posterior mean	one value	yes	0.714
Full posterior	a distribution	yes	Beta(5, 2)

Beta prior choices

Prior	Pseudo-counts	Mean	Effect
Beta(1, 1)	0 likes, 0 dislikes	0.5	flat, mode = MLE
Beta(2, 2)	1 like, 1 dislike	0.5	gentle pull
Beta(20, 20)	19 and 19	0.5	strong pull

8. Common mistakes and myths

Mistake: confusing P(+ | sick) with P(sick | +). 90% sensitivity does not mean a 90% chance of disease after a positive test; here it is 15%.
Mistake: ignoring the base rate. Rare conditions produce mostly false positives even with good tests.
Myth: "Bayesian means subjective guesswork." Priors can be weak, or based on data, and the data wash them out as n grows.
Mistake: counting the same evidence twice. The second test must be a genuinely independent measurement; re-running a flawed lab step is not.
Mistake: a zero-probability prior. It can never be corrected.
Myth: "MAP and MLE are the same." They agree only with a flat prior.
Mistake: reading a credible interval as a confidence interval. They are defined differently (see 1.41).

9. Real-life and AI uses

Medical screening and diagnosis: combining test results with base rates.
Naive Bayes spam filters: P(spam given words) from word likelihoods and a prior (D2L 22.9).
Weight decay and regularisation: a Gaussian prior on weights is the L2 penalty.
Bayesian optimisation of hyperparameters (background).
Bayesian A/B testing: the probability that variant B beats A (background).
Recommender cold start: a prior for new items with few ratings.

10. Self-check questions (with answers)

State Bayes' rule. P(H | D) = P(D | H) P(H) ÷ P(D).
Out of 10,000 people with 1% prevalence, 90% sensitivity, 5% false positives, how many positives? 90 + 495 = 585.
What is P(sick | +)? 90 ÷ 585 = 0.1538, about 15%.
Why is it so low although the test is 90% sensitive? The healthy are 99 times more numerous, so false positives (495) outnumber true ones (90).
After a second independent positive, what is the posterior? About 0.766, roughly 77%.
Update Beta(2, 2) with 3 likes and 0 dislikes. Beta(5, 2), with mean 5/7 = 0.714.
After a further 14 likes and 6 dislikes? Beta(19, 8), mean 0.704, mode 0.72.
Why is a Bayesian answer of 0.714 better than the MLE of 1.00 after 3 of 3? The MLE claims dislikes are impossible; the prior makes the estimate more cautious.
How does MAP with a Gaussian prior relate to deep learning? It equals the loss plus an L2 penalty, which is weight decay.
What happens to the prior as data grow? Its influence fades, and the posterior approaches the MLE.

11. Key takeaways, go deeper, next

Key takeaways

Posterior ∝ likelihood × prior.
Always check the base rate before trusting a test result.
Beta updating is counting: add successes to a and failures to b.
Posterior means sit between the prior mean and the MLE.
Weight decay is MAP with a Gaussian prior.
Yesterday's posterior is today's prior.

Go deeper

Try different Beta priors on the cafe data and watch how quickly they converge.
Read Bishop 2.1.1 on the beta distribution and sequential updating.
Read Goodfellow et al. 5.6 on Bayesian statistics and MAP.

Next on the roadmap: 1.39 The central limit theorem.

12. Sources and further reading

Bishop, Pattern Recognition and Machine Learning (2006), 2.1.1 The beta distribution (Binary variables)
Goodfellow, Bengio, Courville, Deep Learning (2016), 5.6 Bayesian Statistics and 5.6.1 Maximum A Posteriori Estimation
Zhang, Lipton, Li, Smola, Dive into Deep Learning (2023), 22.9 Naive Bayes
Zhang, Lipton, Li, Smola, Dive into Deep Learning (2023), 22.7 Maximum Likelihood
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 3: Probability and Information Theory

Accuracy note: The medical test (1% prevalence, 90% sensitivity, 5% false positive rate) and the cafe taster counts are illustrative, and every number was recomputed in Python. The Beta-update rule, posterior mean lying between prior mean and MLE, and MAP with a Gaussian prior equal to weight decay come from the cited books. The history of Bayes and Laplace, Bayesian optimisation and Bayesian A/B testing are background and are not drawn from the listed sources. The 95% credible interval for Beta(19, 8), about 0.52 to 0.86, was computed with SciPy.

Bayesian updating: prior to posterior: study notes