1.09 Conditional probability and Bayes' theorem: study notes

Learn AI by Story

1. The idea in one sentence

Conditional probability asks "how likely is A, given that I already know B?", and Bayes' theorem is the rule that lets you flip it around: it turns "how likely is this evidence if the thing is true?" into "how likely is the thing, now that I have seen the evidence?".

2. The story from the video

Meera takes a medical test for a rare disease and it comes back positive. She panics. But only 1 in 100 people (1%) have the disease, the test catches 90% of sick people, and it wrongly raises the alarm for 10% of healthy people. Imagine 1,000 people: 10 are sick and 990 are healthy. Of the sick, 9 test positive. Of the healthy, 99 test positive by mistake. So 108 people test positive, and only 9 of them are really sick. Meera's real chance of being sick is 9 out of 108, about 8.3%, far lower than the 90% she feared. A second, independent positive test lifts it to about 45%. Spam filters and many AI systems use exactly this kind of updating.

3. Key definitions (glossary)

Probability. A number from 0 to 1 that measures how likely an event is. Plain English: 0 means impossible, 1 means certain, 0.5 means a coin flip.

Event. A set of outcomes we care about, such as "the test is positive" or "the email is spam". Plain English: a yes/no statement about what happened.

Joint probability, P(A and B). The probability that both A and B happen together. Plain English: the share of all cases where both things are true.

Marginal probability, P(A). The probability of A on its own, ignoring everything else. It is called "marginal" because in a table of counts it is found by adding up a row or column and writing it in the margin. Plain English: the overall rate.

Conditional probability, P(A | B). The probability of A given that B is known to have happened. Formula: P(A | B) = P(A and B) / P(B), valid when P(B) > 0. Plain English: shrink the world to only the cases where B is true, then ask how often A happens inside it.

Product rule. P(A and B) = P(A | B) × P(B). Plain English: to have both, first have B, then have A within B.

Independence. A and B are independent if knowing one tells you nothing about the other. In conditional language: P(A | B) = P(A). Equivalent: P(A and B) = P(A) × P(B). Plain English: learning B does not change your belief about A.

Law of total probability. If events B1, B2, ... split all possibilities into non-overlapping groups, then P(A) = P(A | B1) × P(B1) + P(A | B2) × P(B2) + ... Plain English: the overall rate is a weighted average of the rates inside each group.

Prior, P(H). Your belief in a hypothesis H before seeing the new evidence. Plain English: what you believed first (for Meera, 1%).

Likelihood, P(E | H). The probability of seeing the evidence E if the hypothesis H were true. Plain English: how well the hypothesis predicts the evidence (for Meera, 0.9).

Evidence (marginal likelihood), P(E). The overall probability of seeing the evidence across all hypotheses. Plain English: how common the evidence is in general (for Meera, 0.108).

Posterior, P(H | E). Your updated belief after seeing the evidence. Plain English: what you believe now.

Sensitivity (true positive rate, recall). P(positive | sick). Plain English: of the sick people, what fraction does the test catch? Here 90%.

Specificity (true negative rate). P(negative | healthy). Plain English: of the healthy people, what fraction does the test correctly clear? Here 90%.

False positive rate. P(positive | healthy) = 1 − specificity. Plain English: how often a healthy person gets a false alarm. Here 10%.

False negative rate. P(negative | sick) = 1 − sensitivity. Plain English: how often a sick person is missed. Here 10%.

Precision (positive predictive value). P(sick | positive). Plain English: of everyone who tests positive, what fraction is truly sick? Here about 8.3%.

Base rate (prevalence). How common the thing is in the population before any test. Plain English: the starting rate (here 1%).

Base-rate fallacy. The mistake of ignoring the base rate and focusing only on the test's accuracy. Plain English: forgetting how rare the thing was to begin with.

Bayesian updating. Using the posterior from one piece of evidence as the prior for the next. Plain English: yesterday's conclusion becomes today's starting point.

Naive Bayes. A classifier that applies Bayes' theorem while assuming each feature (for example each word) is independent given the class. Plain English: multiply the clues together as if they were unrelated.

4. Step-by-step explanation

The easiest way to think about probability is as a share of a crowd. If 10 of 1,000 people are sick, then P(sick) = 10/1,000 = 0.01. Counting people is almost always safer than juggling decimals, which is why the video used 1,000 imaginary people. This style is called natural frequencies.

Step 2: Joint and marginal probability from a table

Fill in a table of the 1,000 people from the story.

	Test positive	Test negative	Row total
Sick	9	1	10
Healthy	99	891	990
Column total	108	892	1,000

Each inner cell divided by 1,000 is a joint probability. For example, P(sick and positive) = 9/1,000 = 0.009. The totals in the margins give marginal probabilities: P(sick) = 10/1,000 = 0.01 and P(positive) = 108/1,000 = 0.108.

Step 3: Conditioning means shrinking the world

To find P(sick | positive), throw away everyone who tested negative. Only the column of 108 positives remains. Inside that column, 9 are sick. So the answer is 9/108 ≈ 0.0833. In formula form, P(sick | positive) = P(sick and positive) / P(positive) = 0.009 / 0.108 = 0.0833. The formula is just the table counting written with probabilities. Dividing by P(B) is the "shrinking the world" step: it rescales the new, smaller world so it again adds up to 1.

Step 4: Why the order matters

P(positive | sick) and P(sick | positive) are different questions with different denominators. The first divides by the 10 sick people: 9/10 = 90%. The second divides by the 108 positive people: 9/108 = 8.3%. The test's quoted accuracy is the first number. Meera needs the second. Mixing them up is the single most common error in this topic.

Step 5: Deriving Bayes' theorem

Start from the product rule, written two ways, because "A and B" is the same event as "B and A":

P(A and B) = P(A | B) × P(B)

P(A and B) = P(B | A) × P(A)

Both right-hand sides equal the same thing, so set them equal: P(A | B) × P(B) = P(B | A) × P(A). Divide both sides by P(B):

P(A | B) = P(B | A) × P(A) / P(B)

That is Bayes' theorem. In the disease language: posterior = likelihood × prior / evidence.

Step 6: Getting the evidence term with total probability

The denominator P(positive) is often not given directly. Split the population into the two groups, sick and healthy, and use the law of total probability:

P(positive) = P(positive | sick) × P(sick) + P(positive | healthy) × P(healthy) = 0.9 × 0.01 + 0.1 × 0.99 = 0.009 + 0.099 = 0.108.

This matches the 108 positives out of 1,000 we counted. The two pieces are exactly "true alarms" (0.009) and "false alarms" (0.099). False alarms outnumber true alarms 11 to 1, because healthy people are 99 times more common than sick ones, and even a 10% false alarm rate applied to 990 people (99) swamps 90% applied to 10 people (9).

Step 7: Putting it together

P(sick | positive) = 0.9 × 0.01 / 0.108 = 0.009 / 0.108 = 0.0833, about 8.3%. The prior was 1%, so the positive test multiplied our belief by roughly 8.3, but the starting point was so low that it only reached 8.3%.

Step 8: Updating again with a second test

Now Meera takes a second, independent test of the same kind and it is also positive. Her new prior is the old posterior, 0.0833, and the new "healthy" share is 1 − 0.0833 = 0.9167.

P(sick | second positive) = 0.9 × 0.0833 / (0.9 × 0.0833 + 0.1 × 0.9167) = 0.075 / (0.075 + 0.09167) = 0.075 / 0.16667 = 0.45.

About 45%. Here "independent" means the second test's errors are not tied to the first test's errors given her true state. If both tests share the same flaw (for example, a substance in her blood that confuses that kind of test), the second result adds much less. This is the "why" of repeating tests that use different methods.

Step 9: The odds shortcut

There is a tidy way to see repeated updating. Odds are "sick : healthy". Prior odds are 1 : 99. A positive result has a likelihood ratio of 0.9 / 0.1 = 9, so each positive multiplies the odds by 9. After one positive: 9 : 99, which is 1 : 11, or 1/12 = 8.3%. After two: 81 : 99, which is 81/180 = 45%. After three: 729 : 99, which is 729/828 ≈ 88%. This works because the evidence term P(E) cancels when we compare the two hypotheses.

5. Everything the video skipped

Prosecutor's fallacy

This is the base-rate fallacy dressed in courtroom clothes. It confuses P(evidence | innocent) with P(innocent | evidence). Illustrative numbers: suppose a DNA profile would match a random innocent person with probability 1 in 1,000,000, and a city has 5,000,000 people. We would expect about 5 innocent people to match purely by chance, plus the real culprit, so about 6 matches. With no other evidence, the chance that a particular matching person is guilty is about 1 in 6, roughly 17%, not 99.9999%. P(A | B) is not P(B | A).

Precision depends on the base rate

The same test gives very different precision in different populations. Use a test with 90% sensitivity and 10% false positive rate on 1,000 people:

Base rate	Sick	Healthy	True positives	False positives	Precision
0.1%	1	999	0.9	99.9	about 0.9%
1%	10	990	9	99	about 8.3%
10%	100	900	90	90	50%
50%	500	500	450	50	90%

(The 0.1% row uses fractional people, which is fine as an average.) This is why doctors screen high-risk groups and why "test everyone" programs for very rare conditions produce mostly false alarms.

Sensitivity versus specificity trade-off

Making a test more eager (lower threshold) catches more sick people, which raises sensitivity, but also raises false alarms, which lowers specificity. No test escapes this trade-off; you choose a point depending on the cost of a miss versus the cost of a false alarm. In AI this is the precision-recall trade-off and the ROC curve.

Independence in conditional language

Events A and B are independent when P(A | B) = P(A). A die example: the chance of rolling a 6 is 1/6, and knowing the roll was even changes it to 1/3, so "six" and "even" are not independent. But "six" on the first roll and "six" on the second are independent: P(second six | first six) = 1/6. A common slip is to treat events as independent when they are not (this is exactly the risk in naive Bayes and in two "independent" medical tests).

Conditional independence

Two things can be dependent overall but independent once you know a third. Two words in an email, "free" and "winner", tend to appear together overall, but once you know the email is spam, the naive Bayes model pretends they are unrelated. That pretend-independence is the "naive" part.

Bayesian versus frequentist views

A frequentist defines probability as long-run frequency: if you repeated the situation endlessly, what fraction of times does the event occur? A Bayesian defines probability as a degree of belief that is updated by evidence. Both agree on the maths of Bayes' theorem; they disagree on whether it is sensible to put a probability on a one-off claim such as "this hypothesis is true". Bayesians must choose a prior, which critics call subjective; Bayesians reply that a weak prior is soon overwhelmed by data.

History

Thomas Bayes, an English minister, worked out the idea in the 1700s, and his friend Richard Price published it after Bayes died (1763). Pierre-Simon Laplace independently developed and popularised it later. For much of the 20th century frequentist methods dominated statistics, and Bayesian methods returned strongly once computers could handle the heavy calculations.

Limitations and trade-offs

The answer is only as good as the inputs. If the prior or the false positive rate is wrong, the posterior is wrong.
Real tests often have different error rates for different people, so one fixed number is a simplification.
The independence assumption in naive Bayes is usually false, so its probabilities are overconfident, though its ranking of classes is often still good.
With many hypotheses, computing P(E) exactly becomes expensive, which is why approximate methods exist.

Roadmap connections

Back: 1.08 Probability basics gave us events, the sum to 1, and the multiplication of independent events, which are the ingredients here. Forward: 1.10 Distributions describes how probability spreads over many outcomes, and Bayes' theorem later works on whole distributions (a prior distribution turning into a posterior distribution). 1.11 gives the mean and variance of those distributions, 1.12 shows why more samples sharpen estimates, and 1.13 Entropy measures how uncertain a belief is.

6. Worked examples

Worked example 1: Meera with a changed test

Suppose a different test for the same 1% disease has sensitivity 95% and a false positive rate of 5%. What is P(sick | positive)?

Take 10,000 people for easy whole numbers (illustrative). Sick: 1% of 10,000 = 100. Healthy: 9,900.
True positives: 95% of 100 = 95. Missed: 5.
False positives: 5% of 9,900 = 495. Correctly cleared: 9,405.
All positives: 95 + 495 = 590.
P(sick | positive) = 95/590 = 0.161, about 16.1%.

Check with the formula: 0.95 × 0.01 / (0.95 × 0.01 + 0.05 × 0.99) = 0.0095 / (0.0095 + 0.0495) = 0.0095 / 0.059 = 0.161. Same answer. Lesson: halving the false alarm rate roughly doubled the precision, because false alarms were the dominating part.

Worked example 2: A factory with two machines

Illustrative numbers. Machine A makes 60% of a factory's items and 2% of its items are defective. Machine B makes 40% of the items and 5% are defective. An inspector picks a random defective item. What is the chance it came from machine A?

Prior: P(A) = 0.6, P(B) = 0.4.
Likelihoods: P(defect | A) = 0.02, P(defect | B) = 0.05.
Evidence by total probability: P(defect) = 0.02 × 0.6 + 0.05 × 0.4 = 0.012 + 0.020 = 0.032.
Posterior: P(A | defect) = 0.012 / 0.032 = 0.375, so 37.5%.
P(B | defect) = 0.020 / 0.032 = 0.625, so 62.5%. The two add to 1, as they must.

Even though machine A makes more items, a defective item is more likely to come from B. B's higher defect rate outweighs A's higher volume. As a counting check, take 10,000 items: A makes 6,000 with 120 defects, B makes 4,000 with 200 defects. Defects total 320, and 120/320 = 0.375.

Worked example 3: A naive Bayes spam filter

Illustrative numbers. Of all email, 40% is spam and 60% is not spam ("ham"). Word probabilities learned from past mail:

Word	Chance the word appears in spam	Chance the word appears in ham
free	0.30	0.02
winner	0.20	0.01
meeting	0.05	0.25

Email 1 contains "free" and "winner". Score each class by prior × product of word likelihoods (assuming the words are independent given the class):

Spam score: 0.4 × 0.30 × 0.20 = 0.024.
Ham score: 0.6 × 0.02 × 0.01 = 0.00012.
Total (the evidence): 0.024 + 0.00012 = 0.02412.
P(spam | email) = 0.024 / 0.02412 = 0.995, about 99.5%.

Email 2 contains "free" and "meeting":

Spam score: 0.4 × 0.30 × 0.05 = 0.006.
Ham score: 0.6 × 0.02 × 0.25 = 0.003.
Total: 0.009. P(spam | email) = 0.006 / 0.009 = 0.667, about 67%.

Email 2 is ambiguous, and a filter would send it to spam only if its threshold is below 67%. Real filters use thousands of words and add small amounts to avoid zero probabilities (smoothing). They also add logarithms instead of multiplying tiny numbers, which prevents the numbers underflowing.

7. Comparison tables and diagrams

Quantity	Question it answers	Meera's value
P(sick)	How common is the disease?	1%
P(positive given sick)	Does the test catch sick people?	90%
P(positive given healthy)	How often are healthy people alarmed?	10%
P(positive)	How common are positive results overall?	10.8%
P(sick given positive)	If I test positive, am I sick?	about 8.3%

A text diagram of the 1,000 people:

1,000 people
├── 10 sick (1%)
│    ├── 9 test positive   (true alarm)
│    └── 1 tests negative  (missed)
└── 990 healthy (99%)
     ├── 99 test positive  (false alarm)
     └── 891 test negative (cleared)

Positives = 9 + 99 = 108.  Sick among positives = 9/108.

Prior, likelihood, posterior at a glance:

Term	Symbol	Role	Meera
Prior	P(H)	belief before evidence	0.01
Likelihood	P(E given H)	how well H predicts E	0.9
Evidence	P(E)	how common E is overall	0.108
Posterior	P(H given E)	belief after evidence	0.0833

8. Common mistakes and myths

"A 90% accurate test means 90% chance I'm sick." No. 90% is P(positive | sick). You need P(sick | positive), which also depends on the base rate.
Ignoring the base rate. Rare conditions produce mostly false alarms, even for good tests.
Swapping P(A | B) and P(B | A). Most dogs are not Labradors, but a Labrador is certainly a dog; the two directions are very different.
Assuming two tests are independent when they share a flaw. Repeating the identical test on the same blood sample may repeat the same error.
Adding probabilities that should be multiplied, or vice versa. For independent events, "both" multiplies; "either, if exclusive" adds.
Thinking the prior is "just opinion" and so useless. Priors summarise real background knowledge; ignoring them is itself a choice, and a poor one.
Believing a positive test is meaningless. It is not: Meera's chance rose from 1% to 8.3%. It is useful evidence, just not proof.
Forgetting that probabilities must add to 1 across the full set of hypotheses. A quick sum check catches many arithmetic errors.

9. Real-life and AI/ML uses

Medical screening. Doctors combine test results with age, symptoms and risk factors (the prior) before deciding on follow-up tests.
Spam filtering. Naive Bayes filters, popular since the late 1990s and early 2000s, classify mail using word probabilities as in the example above.
Text classification. Sentiment analysis, topic labelling and language detection are common naive Bayes tasks because the method trains fast with little data.
Fraud detection. A transaction pattern that is rare in genuine use but common in fraud shifts the probability of fraud; the base rate of fraud keeps the alarm rate in check.
Self-driving and robotics. Bayesian filters (such as the Kalman filter and particle filters) repeatedly update the belief about where a robot is as new sensor readings arrive.
Search and rescue. Bayesian search theory updates the map of where a lost object probably is each time an area is searched without success.
Machine learning in general. Training a model can be seen as updating a prior over parameters into a posterior using data. Even when not done explicitly, regularisation is often interpreted as a prior. Language models are also built on conditional probability: they estimate P(next word | the words so far).
Evaluation of classifiers. Precision, recall and the false positive rate all come from the same table of true and false positives and negatives; a rare positive class makes precision hard to keep high.

10. Self-check questions

Write the definition of P(A | B) in words and symbols. It is the probability that A happens given B has happened: P(A | B) = P(A and B) / P(B), when P(B) > 0.
In the story, how many of the 1,000 people test positive, and how many of those are sick? 108 test positive (9 + 99); 9 of them are sick.
Compute P(sick | positive) for Meera. 9/108 = 0.0833, about 8.3%.
Why is P(positive | sick) different from P(sick | positive)? They divide by different groups: the first divides by the 10 sick people (9/10 = 90%), the second by the 108 positives (9/108).
Derive Bayes' theorem from the product rule. P(A and B) = P(A | B) P(B) = P(B | A) P(A); dividing by P(B) gives P(A | B) = P(B | A) P(A) / P(B).
Use the law of total probability to find P(positive). 0.9 × 0.01 + 0.1 × 0.99 = 0.009 + 0.099 = 0.108.
If the base rate rises to 10% with the same test, what is the precision? Of 1,000 people, 100 are sick and 900 healthy; 90 true and 90 false positives; precision = 90/180 = 50%.
What does it mean for A and B to be independent, in terms of conditionals? P(A | B) = P(A): knowing B does not change the chance of A.
A second independent positive test follows the first. What is Meera's new probability? 0.9 × 0.0833 / (0.9 × 0.0833 + 0.1 × 0.9167) = 0.075 / 0.16667 = 0.45, about 45%.
What is the prosecutor's fallacy? Treating P(evidence | innocent) as if it were P(innocent | evidence), ignoring how many people could match by chance.

11. Key takeaways, go deeper, and what is next

Key takeaways

Conditional probability means shrinking the world to the cases where B is true, then asking how often A happens there.
Bayes' theorem: posterior = likelihood × prior / evidence.
The evidence term comes from the law of total probability.
Rare things stay rare after one imperfect test, which is the base-rate lesson from Meera's 8.3%.
Each new independent piece of evidence updates the previous posterior, and odds multiply by the likelihood ratio.
Never treat P(A | B) as P(B | A).
Counting with natural frequencies (1,000 people) is the safest way to avoid mistakes.

Go deeper

Build your own table of 10,000 people for a different test and compare its precision to the formula.
Try the odds form of Bayes' rule with three or four tests in a row.
Look up naive Bayes classification and try the smoothing idea on a tiny set of emails.
Read about the Bayesian versus frequentist debate and the idea of a conjugate prior.
Read about the precision-recall curve and the ROC curve for classifiers.

Next roadmap ID: 1.10 Distributions (uniform, Bernoulli, normal).

Learn AI by Story: one concept, one story, every day.

Conditional probability and Bayes' theorem: study notes