1.34 Joint, marginal and independent events: study notes

Learn AI by Story

The idea in one sentence

A joint probability describes two things happening together, a marginal probability is what you get by summing the other thing away, and two events are independent only when the joint equals the product of the marginals in every single cell.

The story from the video

Anu runs a tea stall. For one hundred customers she writes down two things: tea or coffee, and morning or evening. The counts are tea and morning 45, tea and evening 15, coffee and morning 15, coffee and evening 25. Adding across a row or down a column gives the totals in the margin: 60 percent of customers take tea and 60 percent come in the morning. Zooming into the morning column, 45 of 60 chose tea (0.75), but in the evening only 15 of 40 did (0.375). Since 0.6 times 0.6 is 0.36 and not 0.45, drink and time of day are dependent. Drink against payment method is different: 70 percent pay by phone, 0.6 times 0.7 is 0.42, and every cell matches, so those two are independent. (All the stall data in this note is illustrative.)

Key definitions (glossary)

Sample space. The set of all outcomes an experiment can produce. For one customer's drink it is {tea, coffee}; for two dice it is the 16 ordered pairs. In plain English: the complete list of "what could happen".

Event. A subset of the sample space, a collection of outcomes we care about. "The customer has tea" is an event. Plain English: a yes/no question about the outcome.

Probability. A number from 0 to 1 assigned to every event, with the whole sample space getting 1 and probabilities of non-overlapping events adding up. Plain English: how much of the total belief or long-run frequency an event gets.

Random variable. A function that turns each outcome into a number, so that we can do arithmetic on it. "Number of cones sold" or "1 if tea, 0 if coffee" are random variables. Plain English: a measurement we have not taken yet.

pmf (probability mass function). For a discrete random variable, p(x) = P(X = x). Each value gets a real probability, and the values sum to 1. Our tables are pmfs.

pdf (probability density function). For a continuous random variable, f(x) is a density: the probability of landing in a small interval is about f(x) times its width, and probabilities of ranges come from integrals. A pdf value can exceed 1; a single exact point has probability 0. Plain English: pmf counts mass at points, pdf measures mass per unit length.

Joint probability. P(A, B) or p(x, y): the probability that A and B both happen. In the table, one cell. Plain English: both together.

Marginal probability. The probability of one variable alone, obtained by summing the joint over the other: p(x) = sum over y of p(x, y). The name comes from writing those sums in the margin of the grid (Goodfellow et al., 3.4). Plain English: forget the other variable.

Sum rule (marginalisation). The rule just stated: p(x) is the sum over all y of p(x, y), and in the continuous case the integral of f(x, y) over y. Plain English: add up every way the other thing could have turned out.

Conditional probability. P(A given B) = P(A, B) / P(B), defined when P(B) is greater than 0. Plain English: restrict attention to the cases where B happened and re-measure A inside that smaller world.

Product rule and chain rule. The product rule says P(A, B) = P(A given B) times P(B). Applied repeatedly, the chain rule says p(x1, ..., xn) = p(x1) times p(x2 given x1) times p(x3 given x1, x2) and so on. Plain English: any joint can be built one conditional at a time. This is exactly how a language model scores a sentence.

Bayes' theorem. P(A given B) = P(B given A) times P(A) / P(B), where the denominator comes from the sum rule: P(B) is the sum over A of P(B given A) P(A). Plain English: it flips the direction of conditioning, letting you go from "how likely is the evidence given the cause" to "how likely is the cause given the evidence".

Independence. A and B are independent if P(A, B) = P(A) P(B), equivalently P(A given B) = P(A) when P(B) is positive. For random variables the equation must hold for all pairs of values (Penn State STAT 414, 17.1). Plain English: learning B tells you nothing about A.

Conditional independence. X and Y are conditionally independent given Z if p(x, y given z) = p(x given z) p(y given z) for every value of z (Goodfellow et al., 3.7; D2L 2.6.4). Plain English: once you know Z, X and Y carry no extra information about each other, even though they may be dependent when Z is unknown.

Mutually exclusive events. A and B cannot both happen: P(A, B) = 0. Plain English: they block each other. If both have positive probability they are never independent.

i.i.d. Independent and identically distributed: every sample is drawn from the same distribution, and the draws do not influence each other, so the joint of n samples is the product of n identical factors. Plain English: each new data point is a fresh roll of the same die.

Step-by-step explanation

Step 1. Build the joint table. Count how often each combination occurs and divide by the total (100). Tea and morning 0.45, tea and evening 0.15, coffee and morning 0.15, coffee and evening 0.25. The four cells sum to 1.00, which is the first sanity check you should always run.

Step 2. Marginalise. Sum across a row to remove time of day: P(tea) = 0.45 + 0.15 = 0.60, so P(coffee) = 0.40. Sum down a column to remove drink: P(morning) = 0.45 + 0.15 = 0.60, P(evening) = 0.40. These totals live in the margins.

Step 3. Condition. To ask what morning customers drink, keep only the morning column (total 0.60) and divide each entry by it: P(tea given morning) = 0.45 / 0.60 = 0.75, P(coffee given morning) = 0.15 / 0.60 = 0.25. For evening: P(tea given evening) = 0.15 / 0.40 = 0.375. Conditioning renormalises a column so it sums to 1 again.

Step 4. Test independence. If drink and time were independent, each cell would equal the product of its margins: tea and morning would be 0.6 times 0.6 = 0.36. The actual cell is 0.45. One failing cell is enough to prove dependence. Equivalent test: P(tea given morning) = 0.75 is not P(tea) = 0.60.

Step 5. Test the payment table. Tea and phone 42, tea and cash 18, coffee and phone 28, coffee and cash 12. P(tea) = 0.6, P(phone) = 0.7. Expected cells under independence: 0.6 times 0.7 = 0.42, 0.6 times 0.3 = 0.18, 0.4 times 0.7 = 0.28, 0.4 times 0.3 = 0.12. All four match, so independent. Also P(phone given tea) = 42/60 = 0.7 = P(phone), the same as P(phone given coffee) = 28/40 = 0.7.

Step 6. Remember that margins hide the joint. An independent table with the same margins as the drink-and-time table would be 36, 24, 24, 16. The real table is 45, 15, 15, 25. The margins (0.6, 0.4, 0.6, 0.4) are identical; the dependence is entirely in how the joint is arranged.

Everything the video skipped

Continuous versions. For two continuous variables there is a joint density f(x, y). Marginalising becomes an integral: f(x) = integral of f(x, y) dy over all y (D2L 22.6.1.9). The conditional density is f(x given y) = f(x, y) / f(y). Independence is f(x, y) = f(x) f(y) for all x and y. Integrals are the subject of roadmap ID 1.27. A pdf is not a probability, so the "cells" are infinitesimal areas.

Conditional independence in more depth. Independence and conditional independence are different properties. Two symptoms (cough and fever) may be dependent in the population because both are driven by a disease, yet independent given the disease status. This is the structure behind naive Bayes: features are assumed conditionally independent given the class, so p(features given class) factorises into a product of one-feature terms. The assumption is rarely exactly true, but it makes the model tiny and trainable from little data. The reverse also happens: two independent causes can become dependent once you condition on their common effect.

Simpson-style surprises (qualitative). Relationships in aggregated tables can reverse when you split the data by a third variable. A treatment can look better in each subgroup and worse overall if the groups are very different in size. The lesson is the same as the video's third trap: margins do not determine the joint, and conditioning on a hidden variable can change the story.

Independence of three or more events. Pairwise independence is not full (mutual) independence. For three events you need the product rule to hold for every pair and for the triple. A classic construction with two fair coin flips and the event "the two flips match" gives three events that are pairwise independent but not mutually independent.

Mutually exclusive is not independent. If A and B cannot both happen and both have positive probability, then P(A, B) = 0 while P(A) P(B) is positive, so they are dependent: learning that A happened tells you B did not.

E[XY] = E[X] E[Y] for independent variables. When X and Y are independent, the expectation of the product factorises, which forces covariance to zero. The converse is false, as the next video shows. See 1.35.

Variance of sums. For independent variables Var(X + Y) = Var(X) + Var(Y); in general there is an extra 2 Cov(X, Y) term (Deisenroth et al., 6.4.4). Independence is what makes the extra term vanish. Covariance and the covariance matrix arrive in 1.35 and 1.36.

Bayesian updating and later modules. Bayes' theorem is the engine of roadmap ID 1.38, Bayesian updating: the posterior is proportional to likelihood times prior, and the denominator is a marginal obtained by the sum rule. The chain rule powers autoregressive language models, and conditional independence structures show up in naive Bayes and graphical models in later modules. Maximum likelihood (1.37) uses the i.i.d. assumption to turn a joint over a dataset into a product, then a sum of logs.

Correlation as geometry. For random variables, correlation behaves like the cosine of an angle between centred variables, and independent variables are orthogonal in that sense (Deisenroth et al., 6.4.6). Independence is stronger than orthogonality.

Worked example 1: two four-sided dice

Roll two fair four-sided dice. Let X be the first and Y the second.

The sample space has 16 equally likely ordered pairs, so p(x, y) = 1/16 for each.
Marginal: p(x) = sum over the four values of y of 1/16 = 4/16 = 1/4. Same for Y.
Independence check: p(x) p(y) = 1/4 times 1/4 = 1/16 = p(x, y) for every one of the 16 cells. Independent (Penn State STAT 414, 17.1).
Now define Z as the larger of the two numbers. P(Z = 4) = 7/16, since seven pairs contain a 4 as the maximum (four with X = 4, plus three more with Y = 4 and X less than 4). Then P(X = 1) = 1/4, but P(X = 1, Z = 4) = 1/16 (only the pair (1, 4)). The product P(X = 1) P(Z = 4) = 1/4 times 7/16 = 7/64 is about 0.109, not 0.0625. So X and the maximum are dependent, even though X and Y are independent. Functions of the same variables can create dependence.
Conditional: P(Z = 4 given X = 1) = (1/16) / (1/4) = 1/4, versus P(Z = 4) = 7/16 unconditionally. Knowing X = 1 lowers the chance that the maximum is 4.

Worked example 2: Bayes' theorem on the tea stall and on a medical test

Part A: the tea stall. We know P(tea given morning) = 0.75, P(morning) = 0.60, P(tea) = 0.60. Bayes: P(morning given tea) = 0.75 times 0.60 / 0.60 = 0.75. Check directly: 45 of the 60 tea drinkers came in the morning, 45/60 = 0.75. Both routes agree. The denominator P(tea) came from the sum rule: 0.45 + 0.15.

Part B: a test for a rare condition (invented numbers, computed exactly). Suppose 1 percent of a population has a condition. A test is positive for 90 percent of people who have it (sensitivity) and for 5 percent of people who do not (false-positive rate). The D2L section 2.6.5 treats an HIV-test example of the same shape.

Joint cells out of 10,000 people: have it and positive 0.01 times 0.90 = 0.009 of the population (90 people); have it and negative 10 people; do not have it and positive 0.99 times 0.05 = 0.0495 (495 people); do not have it and negative 9,405 people.
Marginal of a positive test by the sum rule: P(positive) = 0.009 + 0.0495 = 0.0585 (585 people).
Bayes: P(condition given positive) = 0.009 / 0.0585 = 0.1538, about 15.4 percent.
Meaning: even with a good test, most positives are false alarms because the condition is rare. The margin of the grid (P of positive) is dominated by the huge healthy group. Doctors and ML engineers both fall for this when they confuse P(positive given condition) with P(condition given positive).

Comparison tables

Quantity	Symbol	How to get it from the table	Tea stall value
Joint	P(tea, morning)	Read one cell	0.45
Marginal	P(tea)	Sum the row	0.60
Conditional	P(tea given morning)	Cell divided by column total	0.75
Independence product	P(tea) P(morning)	Multiply the two margins	0.36

Table	Tea and morning	Tea and evening	Coffee and morning	Coffee and evening	Independent
Anu's real drink-and-time counts	45	15	15	25	No
Independent table, same margins	36	24	24	16	Yes
Tea and payment (phone, cash order)	42 phone, 18 cash	not used	28 phone, 12 cash	not used	Yes

Idea	Mutually exclusive	Independent
Meaning	Cannot both happen	One does not change the chance of the other
Joint probability	Exactly 0	Product of the marginals
Learning A happened	Tells you B did not	Tells you nothing about B
Both can be true together?	Only if one event has probability 0	Only if one event has probability 0

Concept	Discrete	Continuous
Single variable	pmf p(x), sums to 1	pdf f(x), integrates to 1
Marginalise	Sum over the other variable	Integrate over the other variable
Independence	p(x, y) = p(x) p(y)	f(x, y) = f(x) f(y)

Common mistakes and myths

Myth: mutually exclusive means independent. It is nearly the opposite: exclusive events with positive probability are dependent.
Checking one cell. Independence needs every cell to match. A single matching cell can happen by coincidence; a single failing cell is a proof of dependence.
Assuming the margins determine the joint. The same margins can come from very different joint tables (36, 24, 24, 16 versus 45, 15, 15, 25).
Confusing P(A given B) with P(B given A). The medical-test example shows how far apart they can be. Use Bayes' theorem and the sum rule for the denominator.
Treating "uncorrelated" as "independent". Zero covariance is weaker; see 1.35.
Assuming independence in data that is clustered or sequential. Neighbouring words, repeated measurements on one patient, and duplicate rows in a dataset are not independent draws.
Forgetting the denominator can be zero. Conditioning on an event of probability 0 is undefined in the discrete formula.
Believing conditional independence implies independence, or the reverse. Neither follows from the other.

Real-life uses

Naive Bayes classifiers assume features are independent given the class and multiply per-feature likelihoods.
The i.i.d. assumption lets us treat a training set as a product of identical factors, which underlies maximum likelihood and most generalisation arguments.
Language models use the chain rule: the probability of a sentence is the product of each word's probability given the words before it.
Medical testing and spam filters use Bayes' theorem and marginalisation to turn evidence into updated beliefs.
A/B testing and surveys rely on independence between respondents and between treatment assignment and outcome.
Marketing and retail use contingency tables like Anu's to ask whether two customer behaviours are related.

Self-check questions

A table has cells 0.2, 0.3, 0.1, 0.4 (in the order A and B, A and not B, not A and B, not A and not B). What is P(A)? Answer: 0.2 + 0.3 = 0.5.
In the same table, what is P(B)? Answer: 0.2 + 0.1 = 0.3.
Are A and B independent? Answer: P(A) P(B) = 0.5 times 0.3 = 0.15, but P(A, B) = 0.2. Not independent.
What is P(A given B)? Answer: 0.2 / 0.3 = 0.667.
State the independence test in one line. Answer: the joint equals the product of the marginals in every cell, equivalently conditioning never changes the probability.
Why are mutually exclusive events with positive probability dependent? Answer: their joint is 0, but the product of positive marginals is positive.
If P(tea) = 0.6 and P(phone) = 0.7 and they are independent, what is P(coffee and cash)? Answer: 0.4 times 0.3 = 0.12.
What does the sum rule do? Answer: it removes a variable by summing (or integrating) the joint over it, producing the marginal.
Why is a pdf value allowed to be larger than 1? Answer: it is a density, probability per unit length; probability comes from integrating over an interval.
What does naive Bayes assume? Answer: that features are conditionally independent given the class, so the likelihood is a product of per-feature terms.

Key takeaways

Joint means both together; marginal means sum out the rest; conditional means renormalise a slice.
Independent means the joint equals the product of the marginals in every cell.
Margins never determine the joint, and exclusive is not independent.
The sum rule, product rule and Bayes' theorem are one family of ideas, and they power naive Bayes, i.i.d. likelihoods and language models.

Go deeper

Build your own 2 by 2 table and try to make it independent by adjusting one cell; notice that the other three are forced to move.
Work the continuous version with a joint density that is constant on a unit square, then on a triangle, and see which one factorises.
Try the extra-cause example: two independent coin flips and the event that they match, and check pairwise versus mutual independence.

Next on the roadmap: 1.35 Expectation rules and covariance.

Sources and further reading

Zhang, Lipton, Li, Smola, Dive into Deep Learning, section 2.6 "Probability and Statistics" (2.6.4 Multiple Random Variables, 2.6.5 HIV test example). https://d2l.ai/chapter_preliminaries/probability.html
Zhang et al., Dive into Deep Learning, section 22.6 "Random Variables" (joint densities, marginals). https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/random-variables.html
Goodfellow, Bengio, Courville, Deep Learning (2016), chapter 3 "Probability and Information Theory": 3.4 Marginal Probability, 3.5 Conditional Probability, 3.7 Independence and Conditional Independence. https://www.deeplearningbook.org/contents/prob.html
Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapter 6 "Probability and Distributions": 6.3 Sum Rule, Product Rule, and Bayes' Theorem; 6.4.5 Statistical Independence. https://mml-book.github.io/book/mml-book.pdf
Penn State STAT 414, 17.1 "Two Discrete Random Variables" (joint pmf, marginal pmf, independence, two four-sided dice). https://online.stat.psu.edu/stat414/book/export/html/723

Accuracy note: the tea-stall and payment data and the medical-test numbers are illustrative; the arithmetic was checked exactly.

Search This Blog

Learn AI by Story

Joint, marginal and independent events: study notes