1.08 Probability basics: study notes

Learn AI by Story

1. The idea in one sentence

Probability is a number between 0 and 1 that measures how likely something is, and the rules of probability tell us how to combine those numbers when several things can happen.

2. The story from the video

Ravi has a bag with 10 marbles: 3 red and 7 blue. He closes his eyes and picks one. How likely is red? Three of the ten marbles are red, so the chance is 3/10 = 0.3. The chance of not red is 1 − 0.3 = 0.7, because every possible outcome together adds up to 1. Then Ravi flips two coins and finds 4 equally likely outcomes, so getting two heads has probability 0.5 × 0.5 = 0.25. Finally he rolls a die: the chance of a 5 or a 6 is 2/6, about 0.33. These same ideas power spam filters, language models and classifiers.

3. Key definitions

Term	Precise definition	Plain-English meaning
Experiment (trial)	A process with an uncertain result that can in principle be repeated.	Doing the thing: pick a marble, flip a coin.
Outcome	One single possible result of the experiment.	One exact way it could turn out.
Sample space	The set of all possible outcomes, usually written S.	The complete list of what could happen.
Event	Any collection of outcomes (a subset of the sample space).	A result you care about, such as "a 5 or a 6".
Probability	A number P(A) from 0 to 1 assigned to an event A, obeying the rules below.	How likely the event is.
Equally likely outcomes	Outcomes that each have the same probability.	A fair draw with no favourites.
Classical probability	P(A) = (number of outcomes in A) / (number of outcomes in S), valid when outcomes are equally likely.	Favourable divided by total.
Complement	The event "A does not happen", written not A, with P(not A) = 1 − P(A).	Everything except A.
Mutually exclusive (disjoint)	Two events that cannot happen together: P(A and B) = 0.	They block each other.
Independent events	Events where one happening does not change the probability of the other: P(A and B) = P(A) × P(B).	They ignore each other.
Union	The event "A or B (or both)", written A or B.	At least one of them happens.
Intersection	The event "A and B both happen".	Both happen together.
With replacement	Drawing where each item is put back before the next draw.	The bag is reset every time.
Without replacement	Drawing where items are not put back.	The bag changes after every draw.
Frequentist view	Probability is the long-run fraction of times an event occurs in repeated trials.	What happens if you repeat it forever.
Bayesian view	Probability is a degree of belief that is updated as evidence arrives.	How confident you are, given what you know.
Odds	The ratio of the chance an event happens to the chance it does not: P / (1 − P).	"3 to 7" instead of "30 percent".
Random variable	A quantity whose value is determined by a random outcome.	A number produced by chance, such as a die roll.
Expected value	The probability-weighted average of a random variable's values.	The long-run average result.
Gambler's fallacy	The false belief that past independent results change future chances.	"It is due." It is not.
Softmax	A function that converts a list of scores into positive numbers that sum to 1.	Scores in, probabilities out.
Statistics	The discipline of learning about a population or process from observed data.	Working backwards from data to the rules.

4. Step-by-step explanation

Step 1: Define the experiment and list what can happen

Before any number, decide what you are doing. Ravi pulls one marble from the bag. The sample space is the 10 individual marbles (or, if we only care about colour, the two colours, but then the colours are not equally likely, so counting marbles is safer).

Step 2: Count favourable outcomes against total outcomes

When every outcome is equally likely, P(event) = favourable outcomes divided by total outcomes. Red marbles: 3. Total marbles: 10. So P(red) = 3/10 = 0.3, which can also be read as 30 percent. Why does this work? If you picked a marble a huge number of times with a fair shake, each marble would come out about equally often, so red would turn up about 3 times in every 10.

Step 3: Respect the boundaries

A probability can never be below 0 or above 1. Zero means impossible, one means certain, and 0.5 is a fair toss-up. If you ever compute 1.3 or −0.2, something is wrong.

Step 4: Everything adds up to 1

Some outcome has to happen. So the probabilities of all outcomes in the sample space add to exactly 1. For the bag: P(red) + P(blue) = 0.3 + 0.7 = 1.

Step 5: Use the complement rule

Because the total is 1, P(not A) = 1 − P(A). So P(not red) = 1 − 0.3 = 0.7. This is a very useful shortcut: when "at least one" is hard to count, count "none" and subtract from 1.

Step 6: Combine independent events by multiplying

Flip two fair coins. The sample space has 4 equally likely outcomes: HH, HT, TH, TT. Exactly one is HH, so P(HH) = 1/4 = 0.25. The same answer comes from multiplying: 0.5 × 0.5 = 0.25. Multiplication works because the second flip is a fresh experiment that ignores the first. In half of the cases the first flip is heads, and in half of those the second is also heads, giving half of a half.

Step 7: Combine mutually exclusive events by adding

A die has six equally likely faces. "Roll a 5" and "roll a 6" cannot both happen on one roll, so they are mutually exclusive. Therefore P(5 or 6) = 1/6 + 1/6 = 2/6 = 1/3 ≈ 0.33.

Step 8: Know when you are not allowed to simply add or multiply

Adding only works for mutually exclusive events (otherwise you double count the overlap). Multiplying only works for independent events (otherwise the second probability has to be adjusted, which is the topic of 1.09). Choosing the right rule is most of the skill.

5. Everything the video skipped

The three axioms

Modern probability rests on three rules from Andrey Kolmogorov (1933). First, every event has probability at least 0. Second, the whole sample space has probability 1. Third, for mutually exclusive events, the probability of the union is the sum of their probabilities. Every other rule in these notes can be derived from those three.

The general addition rule

For any two events, mutually exclusive or not: P(A or B) = P(A) + P(B) − P(A and B). We subtract the overlap because adding P(A) and P(B) counts the overlap twice. When the events are mutually exclusive, the overlap is 0 and the rule collapses to simple addition.

Mutually exclusive versus independent

These two ideas are often confused, and they are close to opposites. Mutually exclusive events cannot both happen: if one occurs, the other is ruled out. Independent events do not influence each other: knowing one happened tells you nothing about the other. Two events with non-zero probabilities that are mutually exclusive are never independent, because learning that A happened tells you for certain that B did not. Example on a die: "roll a 5" and "roll a 6" are mutually exclusive (and dependent). "First coin is heads" and "second coin is heads" are independent (and not exclusive).

Sampling with and without replacement

Back to the marble bag. If Ravi puts the marble back, the draws are independent, and P(two reds in a row) = 3/10 × 3/10 = 0.09. If he keeps the first marble, the second draw has changed: only 9 marbles remain, and 2 of them are red, so P(two reds in a row) = 3/10 × 2/9 = 6/90 = 1/15 ≈ 0.067. Without replacement, the draws are dependent, which is a preview of conditional probability in 1.09.

Frequentist versus Bayesian views

A frequentist says "the probability of heads is 0.5 because in a very long run of flips about half come up heads". A Bayesian says "0.5 describes how strongly I believe heads, and I will change that number when I see evidence". The frequentist view struggles with one-off events (will it rain on my birthday?), where repeating is impossible. The Bayesian view handles such cases naturally and is the heart of 1.09. Both use the same mathematical rules.

Odds versus probability

Odds compare the chance for with the chance against. If P = 0.3, odds are 0.3 : 0.7 = 3 : 7, about 0.43 as a single number. To convert back, odds of a : b mean a probability of a / (a + b). Odds of 3 : 1 correspond to 3/4 = 0.75. Bookmakers speak in odds; machine learning uses the logarithm of odds (log-odds or logit) inside logistic regression.

Expected value

For a random variable, multiply each value by its probability and add. For a fair die: (1 + 2 + 3 + 4 + 5 + 6) × 1/6 = 21/6 = 3.5. Nobody ever rolls 3.5, but over many rolls the average approaches it. Expected value is a long-run average, not a prediction of any single result. It returns in 1.11 as the mean, and in 1.12 as the law of large numbers.

The gambler's fallacy

After five heads in a row, the next fair coin flip is still 50 percent heads. The coin has no memory. The probability of six heads in a row from the start is (1/2)^6 = 1/64, but once five have already happened, only the last flip is still uncertain. Streaks look suspicious but are normal in random sequences.

Probability versus statistics

Probability starts from a known model ("a fair die") and asks what data to expect. Statistics goes the other way: starting from data ("I rolled 20 sixes in 60 rolls"), it asks what model could have produced it. Machine learning lives mostly on the statistics side but uses probability rules to reason about its models.

Softmax in one paragraph

A neural network typically ends with raw scores, which can be negative or larger than 1. The softmax function exponentiates each score and divides by the total, so each output is positive and all outputs sum to 1. A worked example appears in section 6.

History and background

The subject began in 1654, when Blaise Pascal and Pierre de Fermat exchanged letters about how to fairly split the stakes of an unfinished game of chance. Jacob Bernoulli's law of large numbers (published 1713) and Thomas Bayes's theorem (published after his death, 1763) followed. Pierre-Simon Laplace built much of the classical theory around 1812, and Kolmogorov placed it on a rigorous footing in 1933.

Limitations and trade-offs

Counting favourable over total only works for equally likely outcomes, and real life rarely offers that. Probabilities of real events must be estimated from data or judgement, so they can be wrong. Assuming independence is convenient, but real events are often linked, and a wrongly assumed independence can badly misjudge risk.

Where this sits on the roadmap

Probability needs only the earlier arithmetic, but it connects to calculus in AI: training chooses model parameters so that the probabilities a model assigns to real data are high, using the gradient work from 1.06 and 1.07. Next, 1.08 leads directly to 1.09 (conditional probability and Bayes), then 1.10 (distributions), 1.11 (mean, variance, correlation), 1.12 (sampling and the law of large numbers) and 1.13 (entropy).

6. Worked examples

Example 1: Two draws from the marble bag

The bag has 3 red and 7 blue marbles. Draw two without replacement.

P(first red) = 3/10.
Given a red was removed, 2 red remain among 9, so P(second red) = 2/9.
P(two reds) = 3/10 × 2/9 = 6/90 = 1/15 ≈ 0.067.
P(two blues) = 7/10 × 6/9 = 42/90 = 7/15 ≈ 0.467.
P(one red and one blue, in either order) = 1 − 1/15 − 7/15 = 7/15 ≈ 0.467.
Check directly: red then blue is 3/10 × 7/9 = 21/90, blue then red is 7/10 × 3/9 = 21/90, and 21/90 + 21/90 = 42/90 = 7/15. It matches.

With replacement, the same questions give 0.09 for two reds, 0.49 for two blues, and 2 × 0.3 × 0.7 = 0.42 for one of each. Those three also add up to 1 (0.09 + 0.49 + 0.42 = 1.00).

Example 2: Two dice and the complement trick

Roll two fair dice. There are 6 × 6 = 36 equally likely outcomes.

Sum equals 7: the pairs (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) give 6 outcomes, so P = 6/36 = 1/6 ≈ 0.167.
At least one six: counting directly is awkward, so use the complement. No sixes means each die shows 1 to 5, so 5 × 5 = 25 outcomes and P(no six) = 25/36.
P(at least one six) = 1 − 25/36 = 11/36 ≈ 0.306.

Example 3: The general addition rule on one die

Let A = "even number" = {2, 4, 6}, so P(A) = 3/6. Let B = "greater than 3" = {4, 5, 6}, so P(B) = 3/6. The overlap is {4, 6}, so P(A and B) = 2/6. Then P(A or B) = 3/6 + 3/6 − 2/6 = 4/6 ≈ 0.667. Check by listing: A or B = {2, 4, 5, 6}, which is 4 outcomes out of 6. It matches. Simple adding would have given 1, which is clearly wrong.

Example 4: Expected value of a simple game (illustrative numbers)

A game pays 10 dollars with probability 0.3 and costs 2 dollars with probability 0.7. Expected value = 10 × 0.3 + (−2) × 0.7 = 3 − 1.4 = 1.6 dollars per play. Over many plays you would average about 1.60 dollars, even though no single play ever pays exactly that.

Example 5: Softmax with three scores

A tiny classifier outputs scores 2, 1 and 0 for cat, dog and bird (illustrative).

Exponentiate: e² ≈ 7.389, e¹ ≈ 2.718, e⁰ = 1.
Sum: 7.389 + 2.718 + 1 = 11.107.
Divide: cat ≈ 7.389 / 11.107 ≈ 0.665, dog ≈ 2.718 / 11.107 ≈ 0.245, bird ≈ 1 / 11.107 ≈ 0.090.
Check: 0.665 + 0.245 + 0.090 = 1.000.

Raising the cat score to 3 while the others become 1 and −1 gives e³ ≈ 20.086, e¹ ≈ 2.718 and e⁻¹ ≈ 0.368, with total about 23.172, so the probabilities are about 0.867, 0.117 and 0.016. A bigger gap in scores means a more confident output.

7. Comparison tables and diagrams

Idea	Question it answers	Rule
Complement	Chance of not A?	1 − P(A)
Mutually exclusive union	Chance of A or B when they cannot overlap?	P(A) + P(B)
General union	Chance of A or B when they may overlap?	P(A) + P(B) − P(A and B)
Independent intersection	Chance of A and B when they ignore each other?	P(A) × P(B)
Dependent intersection	Chance of A and B when one affects the other?	Needs conditional probability (1.09)

Mutually exclusive	Independent
Cannot both happen	Do not influence each other
P(A and B) = 0	P(A and B) = P(A) × P(B)
Knowing A tells you B is false	Knowing A tells you nothing about B
Roll 5 versus roll 6	Coin one versus coin two

Frequentist	Bayesian
Long-run frequency	Degree of belief
Needs repeatable trials	Works for one-off events
Parameters are fixed unknowns	Parameters get probability distributions

Tree of two draws without replacement (first draw, then second):

Start → Red (3/10) → Red (2/9) or Blue (7/9)

Start → Blue (7/10) → Red (3/9) or Blue (6/9)

Multiply along a branch, add across branches that end the same way.

8. Common mistakes and myths

Adding probabilities of events that overlap. Use the general addition rule.
Multiplying probabilities of events that are not independent, such as draws without replacement.
Believing "mutually exclusive" means "independent". They are nearly opposite ideas.
The gambler's fallacy: thinking a coin is due for tails after a streak of heads.
Assuming all outcomes are equally likely. "Red or blue" is two outcomes but not a 50/50 chance in the bag.
Reading expected value as a guarantee. It is a long-run average, not what happens next.
Mixing up probability and odds. A probability of 0.3 is odds of 3 : 7, not 3 : 10.
Treating a model's softmax output as certainty. A confident wrong answer is still possible; the numbers describe the model's belief, not the truth.
Believing a probability of 0 always means literally impossible. In continuous settings (see 1.10), events of probability 0 can still happen. For this beginner level, treat 0 as impossible.

9. Real-life and AI/ML uses

Weather forecasts state a probability of rain. Insurance companies price policies using the probability of claims. Doctors weigh the probability that a test result is right. Games, lotteries and quality-control sampling all rely on it.

In AI: a spam filter assigns each email a probability of being spam, and moves it to the junk folder above a chosen threshold (illustrative: 0.9). A language model produces a probability for every possible next word, then picks or samples from them, so after "The cat sat on the" it may assign high probability to "mat" and tiny probability to "volcano". An image classifier uses softmax so that its outputs for cat, dog and bird sum to 1. Training objectives such as cross-entropy (see 1.13) reward the model for giving high probability to the correct answer. Generative models, recommendation systems and self-driving perception all reason in probabilities, because the real world is uncertain.

10. Self-check questions

What range of values can a probability take? Answer: from 0 (impossible) to 1 (certain), inclusive.
A bag has 3 red and 7 blue marbles. What is P(red), and what is P(not red)? Answer: 0.3 and 0.7.
What is the probability of two heads on two fair coin flips, and why? Answer: 0.25, because the flips are independent: 0.5 × 0.5 (or 1 of 4 equally likely outcomes).
What is P(5 or 6) on a fair die? Answer: 2/6 = 1/3 ≈ 0.33, added because the events are mutually exclusive.
Give the general addition rule. Answer: P(A or B) = P(A) + P(B) − P(A and B).
What is P(two reds) drawing from the marble bag without replacement? With replacement? Answer: 3/10 × 2/9 = 1/15 ≈ 0.067 without; 0.09 with.
What is the difference between mutually exclusive and independent? Answer: mutually exclusive events cannot both happen; independent events do not affect each other's probability.
What is the expected value of a fair die roll? Answer: 3.5.
After five tails in a row on a fair coin, what is the probability of heads next? Answer: still 0.5; the coin has no memory.
If a model outputs scores 2, 1, 0, roughly what are the softmax probabilities? Answer: about 0.665, 0.245 and 0.090, which sum to 1.

11. Key takeaways, go deeper, and what is next

Key takeaways:

Probability is a number from 0 to 1; all outcomes together total 1.
Favourable divided by total works when outcomes are equally likely.
Complement: P(not A) = 1 − P(A). Exclusive events add; independent events multiply.
Overlaps need the general addition rule; changing bags (no replacement) break independence.
Expected value is a long-run average, and softmax turns scores into probabilities.

Go deeper:

Compute all outcomes of three coin flips and check they sum to 1.
Repeat the marble examples with a different bag, such as 5 red and 5 blue.
Read about the birthday paradox, a famous surprising probability result.
Experiment with softmax on your own scores and watch how gaps change confidence.
Explore the Bayesian versus frequentist debate through a short introductory article.

Next on the roadmap: 1.09 Conditional probability and Bayes' theorem.

Learn AI by Story: one concept, one story, every day.

Probability basics: study notes