1.37 Maximum likelihood estimation: study notes

Learn AI by Story

1. The idea in one sentence

Maximum likelihood estimation (MLE) picks the parameter value under which the data you actually saw would have been most probable: if Ravi makes 7 of 10 free throws, the skill level p̂ = 0.7 explains that result better than any other p does.

2. The story from the video

Ravi practises free throws. One evening he takes 10 shots and makes 7. His friends argue about his "true" skill. One says 50%, because "that is what most players are". Another says 90%, because "he looked hot". Ravi asks a cleaner question: for each possible skill level p, how likely is it that someone with that skill would produce exactly 7 makes and 3 misses, in this order or any order that has the same counts?

He tries values one at a time. With p = 0.5 the chance of the sequence is p⁷(1 − p)³ = 0.000977. With p = 0.6 it is 0.001792. With p = 0.7 it is 0.002224, the biggest of all. With p = 0.8 it falls back to 0.001678, and with p = 0.9 it is only 0.000478. The curve rises, peaks at 0.7, and falls. The peak is the maximum likelihood estimate, p̂ = 0.7, which is simply 7 ÷ 10, the fraction he actually made.

The video then shows a trick. Multiplying many small probabilities gives absurdly tiny numbers (0.5¹⁰⁰⁰ ≈ 9.33 × 10⁻³⁰²), so we take logarithms: the product turns into a sum, and the peak stays in the same place. Finally the story ends with a warning: if Ravi makes 3 of 3 shots, MLE says p̂ = 1, "he never misses", which is silly. The fix is to bring in prior belief, which is the next video (1.38).

3. Key definitions (glossary)

Parameter (θ): a number (or list of numbers) that fixes a model, such as the success chance p of a coin or the mean μ and variance σ² of a Gaussian. Plain English: the dial settings of the model.
Data (D): the observed sample x₁, x₂, …, xₙ. Plain English: what actually happened.
Likelihood L(θ): the probability (or probability density) the model assigns to the observed data, viewed as a function of θ with the data held fixed: L(θ) = P(D | θ). Plain English: how well this dial setting explains what we saw.
Log-likelihood ℓ(θ): ℓ(θ) = ln L(θ). Plain English: the same score, but sums instead of products.
Maximum likelihood estimate θ̂: θ̂ = argmax over θ of L(θ) = argmax of ℓ(θ). Plain English: the dial setting with the highest score.
Estimator: a rule that turns data into a guess for θ. Plain English: the recipe; θ̂ is its output for one data set.
i.i.d.: independent and identically distributed; each observation is drawn independently from the same distribution. Plain English: every data point is a fresh roll of the same die.
Negative log-likelihood (NLL): −ℓ(θ). Plain English: a loss; minimising it is the same as maximising likelihood.
Score equation: ℓ′(θ) = 0. Plain English: where the log-likelihood curve is flat, the top of the hill.
Bias: E[θ̂] − θ. Plain English: does the guess land on target on average?
Consistency: θ̂ gets closer to the true θ as n grows. Plain English: more data, better guess.
Cross-entropy: for labels y and predicted probabilities q, −Σ y ln q. Plain English: the loss that neural classifiers minimise, which equals the NLL of the labels.

4. Detailed explanation, from basics to the full picture

Step 1: Probability versus likelihood

Probability asks "given the dial setting, how likely is each possible data set?" Likelihood asks the reverse question: "given the data, how plausible is each dial setting?" The formula is identical, P(D | θ), but what you vary changes. In probability, θ is fixed and D varies. In likelihood, D is fixed and θ varies. This is why the likelihood curve over p does not have to add up to 1; it is a score for each θ, not a probability distribution over θ. (Making that into a distribution over θ needs a prior: video 1.38.)

Step 2: Write the likelihood for i.i.d. data

If the observations are independent, the probability of all of them is the product of the individual probabilities: L(θ) = P(x₁ | θ) × P(x₂ | θ) × … × P(xₙ | θ). For free throws with a make coded 1 and a miss coded 0, each shot has probability p when it is a make and 1 − p when it is a miss. With 7 makes and 3 misses, L(p) = p⁷(1 − p)³. Strictly, the chance of "7 makes in any order" is C(10, 7) = 120 times larger, but 120 does not depend on p, so it does not move the peak.

Step 3: Take the logarithm

Because ln is increasing, the p that maximises L also maximises ln L. And ln turns products into sums: ℓ(p) = 7 ln p + 3 ln(1 − p). For p = 0.5 this is −6.931, for p = 0.7 it is −6.109, and for p = 0.9 it is −7.645. Same ordering as before, just friendlier numbers. For 1000 fair-coin flips, 0.5¹⁰⁰⁰ ≈ 9.33 × 10⁻³⁰² is almost at the edge of what a computer float can store, while 1000 × ln 0.5 = −693.1 is perfectly ordinary. D2L makes the same point: the (1/2) to the power one billion is far below machine precision, but its log, about −301,029,995.6, fits easily.

Step 4: Find the peak with calculus

Differentiate: ℓ′(p) = 7/p − 3/(1 − p). Set it to zero: 7/p = 3/(1 − p), so 7(1 − p) = 3p, so 7 = 10p, so p̂ = 0.7. In general for k successes in n trials, ℓ′(p) = k/p − (n − k)/(1 − p) = 0 gives p̂ = k/n. The second derivative ℓ″(p) = −7/p² − 3/(1 − p)² is negative everywhere, so the curve is a single hill and this really is a maximum.

Step 5: The Gaussian case

For data x₁, …, xₙ from N(μ, σ²), the log-likelihood is ℓ(μ, σ²) = −(n/2) ln(2πσ²) − Σ(xᵢ − μ)² ÷ (2σ²). Setting the derivative in μ to zero gives μ̂ = (1/n) Σ xᵢ, the sample mean. Setting the derivative in σ² to zero gives σ̂² = (1/n) Σ (xᵢ − μ̂)². So the MLE of the mean is the plain average, and the MLE of the variance divides by n, not n − 1.

Step 6: What the MLE does and does not promise

Under the usual conditions (the true distribution is in the model family and the parameters are identifiable), the MLE is consistent: as n grows, θ̂ converges to the true θ (Goodfellow et al., 5.5.2). It is not always unbiased in small samples. The Gaussian variance MLE is the classic example: it underestimates the true variance by the factor (n − 1)/n (Bishop, 1.2.4).

Step 7: Why machine learning cares

Training a classifier with cross-entropy loss is MLE for the labels. Training a regression model with mean squared error is MLE when the noise is assumed Gaussian (Goodfellow et al., 5.5.1). A language model trained to predict the next token maximises the log-probability of each real next token given the ones before it, which is again MLE.

5. Everything the video skipped

Formulas in simple words

Bernoulli: p̂ = (number of successes) ÷ n. "Use the fraction you saw."
Categorical with K outcomes: p̂ₖ = nₖ ÷ n. "Use each outcome's observed fraction."
Gaussian mean: μ̂ = (1/n) Σ xᵢ. "Use the average."
Gaussian variance: σ̂² = (1/n) Σ (xᵢ − μ̂)². "Average squared distance from the average."
Unbiased variance: s² = (1/(n − 1)) Σ (xᵢ − μ̂)² . "Same, but divide by one less, to cancel the bias."
Regression: if y = wx + noise with noise N(0, σ²), then maximising likelihood over w is the same as minimising Σ (yᵢ − wxᵢ)².

The role of the i.i.d. assumption

The product formula L = Π P(xᵢ | θ) only holds when observations are independent. If the data are time series, or the same customer appears many times, you must model the dependence (for example with the chain rule P(x₁) P(x₂ | x₁) …), or the likelihood will be too confident.

Invariance

If θ̂ is the MLE of θ, then g(θ̂) is the MLE of g(θ) for a function g (background fact). So if p̂ = 0.7 for makes, the MLE for the odds p/(1 − p) is 0.7/0.3 ≈ 2.33, with no new maximisation.

Link to KL divergence

Maximising average log-likelihood of the data is equivalent to minimising the KL divergence from the empirical data distribution to the model distribution (Goodfellow et al., 5.5). The entropy of the data does not depend on θ, so only the cross-entropy term matters. This is why "minimise cross-entropy" and "maximise likelihood" are the same instruction.

Limitations

Small samples: 3 makes of 3 gives p̂ = 1, a wild over-fit (Bishop, 2.1). The MLE assigns zero probability to any future miss.
Zero counts: in a categorical model, an outcome never seen gets p̂ = 0, which breaks any later product that contains it.
Model misspecification: if the true process is not in the model family, the MLE finds the closest member, not the truth.
Complex models: with huge models (neural networks) the MLE can memorise the training set. Regularisation or priors (video 1.38) push back.
The likelihood may have many local maxima, so numerical optimisers can get stuck.
MLE gives one number; it does not by itself say how uncertain that number is. Confidence intervals (video 1.41) do.

Background

The method is usually credited to Ronald Fisher, who popularised it in the early 1900s (background, not from the sources above). The point to remember is the one idea: pick the θ that makes the data least surprising.

Links on the roadmap

1.34 joint, marginal and independent events (why products work), 1.35 expectation and covariance, 1.28 log and softmax, 1.29 log-sum-exp (numerically stable logs of sums), 2.05 loss functions (where cross-entropy and MSE live), and 1.38 Bayesian updating (the cure for the 3-of-3 trap).

6. Worked examples with real numbers

Example A: Ravi's 7 of 10 free throws, by grid and by calculus

Setup: 7 makes, 3 misses, so L(p) = p⁷(1 − p)³.

Evaluate on a grid: L(0.5) = 0.000977, L(0.6) = 0.001792, L(0.7) = 0.002224, L(0.8) = 0.001678, L(0.9) = 0.000478.
The biggest value is at p = 0.7.
Take logs: ℓ(p) = 7 ln p + 3 ln(1 − p), giving ℓ(0.5) = −6.931, ℓ(0.7) = −6.109, ℓ(0.9) = −7.645.
Differentiate: ℓ′(p) = 7/p − 3/(1 − p).
Set ℓ′(p) = 0: 7(1 − p) = 3p, so 10p = 7.
Solve: p̂ = 0.7.

Result: p̂ = 7/10 = 0.7, and the grid and calculus agree.

Example B: Gaussian delivery times

Setup (illustrative): five delivery times in minutes are 28, 31, 30, 35, 26. Assume they are i.i.d. N(μ, σ²).

MLE of the mean: μ̂ = (28 + 31 + 30 + 35 + 26) ÷ 5 = 150 ÷ 5 = 30.
Deviations from 30: −2, 1, 0, 5, −4.
Squared deviations: 4, 1, 0, 25, 16, sum = 46.
MLE variance (divide by n = 5): σ̂² = 46 ÷ 5 = 9.2, so σ̂ ≈ 3.03 min.
Unbiased variance (divide by n − 1 = 4): s² = 46 ÷ 4 = 11.5.
Compare: the MLE is smaller by exactly the factor 4/5, which is (n − 1)/n.

Result: μ̂ = 30 min, σ̂² = 9.2 (MLE) versus 11.5 (unbiased). With big n the two almost coincide.

Example C: A die with six faces

Setup (illustrative): you roll a die 60 times and count faces 1 to 6: 8, 12, 9, 11, 10, 10.

Model: a categorical distribution with probabilities p₁, …, p₆ that add to 1.
Log-likelihood: ℓ = Σ nₖ ln pₖ, maximised under Σ pₖ = 1.
The solution is p̂ₖ = nₖ ÷ 60.
So p̂₁ = 8/60 = 0.133, p̂₂ = 12/60 = 0.200, p̂₃ = 9/60 = 0.150, p̂₄ = 11/60 = 0.183, p̂₅ = p̂₆ = 10/60 = 0.167.

Result: the MLE is just the observed frequency of each face; the die looks slightly lopsided, but 60 rolls is too few to be sure.

Example D: Why logs rescue a computer

Setup: 1000 fair-coin flips, and you want the likelihood of any single exact sequence.

Direct product: 0.5¹⁰⁰⁰ ≈ 9.33 × 10⁻³⁰².
In 64-bit floating point this is already close to the smallest representable size; a few hundred more flips would give exactly 0.
Log version: 1000 × ln 0.5 = −693.1.

Result: the log-likelihood is a harmless number; the raw likelihood is on the edge of disappearing.

7. Comparison tables

Probability versus likelihood

Feature	Probability	Likelihood
Fixed	the parameter θ	the data D
Varies	the data	the parameter
Adds to 1	yes over all data	not over θ
Used for	predicting	estimating

Three common MLEs

Model	Parameter	MLE	Plain English
Bernoulli	p	k ÷ n	fraction of successes
Categorical	pₖ	nₖ ÷ n	observed frequency
Gaussian	μ, σ²	mean, ÷ n variance	average and spread

MLE in ML losses

Assumption	Likelihood	Loss
Labels are categories	categorical	cross-entropy
Noise is Gaussian	Gaussian	MSE
Next token	categorical over vocabulary	token cross-entropy

8. Common mistakes and myths

Myth: "The likelihood is the probability that p = 0.7." No. It is the probability of the data if p were 0.7. It is not a distribution over p.
Mistake: forgetting that logs are only a convenience. The maximiser is the same, but the values differ; do not compare a likelihood with a log-likelihood.
Mistake: dividing by n and calling it unbiased. The MLE variance has bias factor (n − 1)/n.
Myth: "MLE always gives the truth with enough data." Only if the model family contains the truth and the data are i.i.d.
Mistake: trusting 3 of 3. p̂ = 1 says a miss is impossible.
Mistake: maximising a likelihood and reporting it as certainty. A point estimate has no error bar unless you add one.
Mistake: mixing up maximising likelihood with minimising loss. They are the same only when the loss is the negative log-likelihood.

9. Real-life and AI uses

Image and text classifiers: cross-entropy training is MLE of the labels.
Regression and forecasting: MSE training is MLE under Gaussian noise.
Large language models: pre-training maximises the log-probability of each next token.
Spam and click-rate estimation: the fraction of clicks is the Bernoulli MLE.
Sports and quality control: success rates such as Ravi's.
Fitting distributions in finance, medicine and engineering: estimating a mean and a spread from measurements.

10. Self-check questions (with answers)

What does L(p) = p⁷(1 − p)³ describe? The probability of a specific sequence of 7 makes and 3 misses, as a function of the skill p.
Why is p̂ = 0.7 and not 0.5? L(0.7) = 0.002224 beats L(0.5) = 0.000977; 0.7 gives the data their highest probability.
Why take logarithms? Products become sums and tiny numbers become ordinary ones (0.5¹⁰⁰⁰ ≈ 9.33 × 10⁻³⁰² versus −693.1), while the peak stays in the same place.
Derive p̂ for k successes in n trials. ℓ′(p) = k/p − (n − k)/(1 − p) = 0 gives p̂ = k/n.
Does the likelihood curve add up to 1 over p? No; it is a score, not a probability distribution over p.
For delivery times 28, 31, 30, 35, 26, what are μ̂ and the MLE variance? μ̂ = 30 and σ̂² = 46 ÷ 5 = 9.2.
Why does the MLE variance underestimate? It measures deviations from the sample mean, which is closer to the data than the true mean, so it is too small by the factor (n − 1)/n; dividing by n − 1 gives 11.5.
What goes wrong with 3 of 3 makes? p̂ = 1, which says a miss is impossible; this is over-fitting, and a prior fixes it.
How is cross-entropy related to MLE? Minimising cross-entropy over the labels is the same as maximising their likelihood.
Which loss corresponds to Gaussian noise? Mean squared error.

11. Key takeaways, go deeper, next

Key takeaways

Likelihood scores each parameter by how probable it makes the observed data.
The maximum is found by setting the derivative of the log-likelihood to zero.
For coins, categories and Gaussian means, the MLE is the obvious "use what you saw".
Logs prevent underflow and turn products into sums.
Cross-entropy and MSE training are MLE in disguise.
With tiny samples MLE over-fits; priors cure that.

Go deeper

Work out the Poisson MLE (it is the sample mean).
Verify numerically that the Gaussian variance MLE averages (n − 1)/n × σ² with a simulation.
Read D2L 22.7 and Goodfellow et al. 5.5.

Next on the roadmap: 1.38 Bayesian updating: prior to posterior.

12. Sources and further reading

Zhang, Lipton, Li, Smola, Dive into Deep Learning (2023), 22.7 Maximum Likelihood
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 5: Machine Learning Basics (5.4 Estimators, Bias and Variance; 5.5 Maximum Likelihood Estimation)
Bishop, Pattern Recognition and Machine Learning (2006), 1.2.4 The Gaussian distribution and 2.1 Binary variables
Goodfellow, Bengio, Courville, Deep Learning (2016), Chapter 3: Probability and Information Theory
Zhang, Lipton, Li, Smola, Dive into Deep Learning (2023), 22.10 Statistics

Accuracy note: The free-throw numbers (7 of 10, the grid values, the logs) were recomputed in Python. The delivery times and the die counts are illustrative, made up for teaching, and verified arithmetically. The 3-of-3 over-fit, the (n − 1)/n variance bias, consistency conditions, cross-entropy and Gaussian-noise links come from the cited books. The attribution of MLE to Fisher and the invariance property are background facts, not taken from the sources listed.

Maximum likelihood estimation: study notes