1.36 The covariance matrix and the multivariate Gaussian: study notes

Learn AI by Story

The idea in one sentence

A covariance matrix Σ stores every spread and every pairwise co-movement of a cloud of data in one table, and the multivariate Gaussian uses Σ to turn "how far from the mean" into a tilted ellipse of likely points, measured with the Mahalanobis distance instead of the straight-line distance.

The story from the video

Dev measures the height and weight of his classmates. Heights have variance 49 (standard deviation 7 cm), weights have variance 64 (standard deviation 8 kg), and the covariance is 42, so the correlation is 42 / (7 × 8) = 0.75: taller usually means heavier. The class averages 170 cm and 65 kg. Two classmates are both 177 cm tall, one weighing 72 kg and the other 58 kg. Both sit 9.9 units from the mean in a straight line, yet their Mahalanobis distances are 1.02 and 2.65: tall and light is rare in this class. The Gaussian's contours are tilted ellipses whose axes are Σ's eigenvectors, and only 39 percent of a 2-D Gaussian lies inside the 1σ ellipse, not 68 percent. (All class data in this story are illustrative.)

Key definitions

Each term below has a precise definition, then what it means in plain English.

Random vector. A list x = (x₁, …, xₙ) of n random variables measured together on the same outcome. Plain English: one row of your table, such as one student's (height, weight), before you know the values.

Mean vector (μ). The vector of expectations, μ = E[x] = (E[x₁], …, E[xₙ]). Plain English: the centre of the cloud. For the story, μ = (170, 65).

Covariance matrix (Σ). The n × n matrix with entries Σᵢⱼ = Cov(xᵢ, xⱼ) = E[(xᵢ − μᵢ)(xⱼ − μⱼ)]. The diagonal entries are the variances; the off-diagonal entries are covariances. Plain English: a spreadsheet of "how much does each thing vary" (diagonal) and "how much do each pair of things move together" (off-diagonal). Story: Σ = [[49, 42], [42, 64]].

Symmetric matrix. A matrix equal to its transpose, Σᵢⱼ = Σⱼᵢ. Covariance is symmetric because Cov(X, Y) = Cov(Y, X). Plain English: the table reads the same mirrored across the diagonal.

Positive semidefinite (PSD). A symmetric matrix A with vᵀAv ≥ 0 for every vector v. Every covariance matrix is PSD. Plain English: no direction has negative spread. Positive definite (PD) means vᵀAv > 0 for every non-zero v, so every direction has strictly positive spread and the matrix is invertible. The Gaussian density needs Σ to be PD.

Correlation matrix. The covariance matrix rescaled so every variable has standard deviation 1: Rᵢⱼ = Σᵢⱼ / (σᵢ σⱼ). The diagonal is all 1s and every entry lies between −1 and 1. Plain English: the same relationships with units removed. Story: R = [[1, 0.75], [0.75, 1]].

Precision matrix. The inverse of the covariance matrix, Λ = Σ⁻¹ (called β in Goodfellow, Bengio and Courville). Plain English: the "tightness" table. It is the matrix that sits inside the Mahalanobis distance and the exponent of the density.

Determinant and the normalizing constant. det Σ is the product of Σ's eigenvalues and measures the squared "volume" of the cloud. The density must integrate to 1, and a wide cloud spreads the same total probability over more area, so the peak must be lower. That is the job of the factor 1 / sqrt((2π)ⁿ det Σ). Plain English: the determinant is the size correction that keeps the total probability at exactly 1.

Mahalanobis distance. d(x) = sqrt((x − μ)ᵀ Σ⁻¹ (x − μ)). Plain English: distance from the mean measured in the cloud's own units and shape. When Σ = I it equals the ordinary Euclidean distance (Bishop 2.3).

Eigen-decomposition. Because Σ is real and symmetric, Σ = U Λ Uᵀ, where the columns of U are orthonormal eigenvectors and Λ is diagonal with eigenvalues λ₁, …, λₙ ≥ 0. Plain English: every covariance matrix is "rotate, stretch along the axes, rotate back." The eigenvectors are the ellipse's axis directions and the square roots of the eigenvalues are the standard deviations along those axes.

Isotropic covariance. Σ = σ²I: equal variance in every direction and no covariances. Plain English: the cloud is a round ball, and the contours are circles or spheres.

Diagonal covariance. Σ has non-zero entries only on the diagonal. Plain English: each variable has its own spread, but there is no tilt. For a Gaussian, a diagonal Σ means the variables are independent.

Chi-square distribution. The distribution of a sum of squares of k independent standard normal variables, with k degrees of freedom. If x is Gaussian in n dimensions, then d² = (x − μ)ᵀ Σ⁻¹ (x − μ) follows a chi-square distribution with n degrees of freedom. Plain English: it tells you how likely it is that a point lands inside any given Mahalanobis radius.

Cholesky factor. For a PD matrix Σ, the unique lower-triangular matrix L with positive diagonal such that Σ = L Lᵀ. Plain English: a "square root" of the matrix that is cheap to apply, used to turn independent noise into correlated noise.

Standardization and whitening. Standardizing divides each variable by its standard deviation (after subtracting the mean). Whitening goes further and applies a transform W so that the result has covariance I, for example z = L⁻¹(x − μ). Plain English: undo the cloud's tilt and stretch so it becomes a round, unit-size ball.

Step-by-step explanation

Step 1. Describe one variable. For a single variable, a mean and a variance describe a Gaussian bell curve. The mean is the centre and the variance is the width.

Step 2. Describe several variables at once. Stack the variables into a random vector. The centre becomes the mean vector μ, a list of means. The width now has to cover not only each variable's spread but also how the variables move together, which is what the covariance matrix Σ does.

Step 3. Build Σ. Put each variance on the diagonal. Put each covariance off the diagonal. In the story the variances are 49 and 64 and the covariance is 42. The matrix is symmetric, so there are only three distinct numbers.

Step 4. Read the correlation. Divide the covariance by the product of the standard deviations: 42 / (7 × 8) = 0.75. Correlation is unit-free and lies in [−1, 1]; covariance has units (here cm times kg).

Step 5. Measure distance honestly. Ordinary distance treats a centimetre and a kilogram as equal and ignores the tilt. The Mahalanobis distance d² = (x − μ)ᵀ Σ⁻¹ (x − μ) first removes both: the inverse of Σ shrinks directions where the data spread a lot and magnifies directions where the data hardly spread.

Step 6. Turn distance into density. The multivariate Gaussian density in words is: one divided by the square root of ((2π) to the power n, times the determinant of Σ), multiplied by e raised to the power of minus one half times d². Written out:

p(x) = 1 / sqrt((2π)ⁿ det Σ) × exp(−½ d²), with d² = (x − μ)ᵀ Σ⁻¹ (x − μ).

The first factor is the constant that makes the total probability 1. The second factor decays as the Mahalanobis distance grows. Points with the same d have the same density, so the contours are the sets {x : d(x) = c}.

Step 7. See the ellipse. Eigen-decompose Σ. The contour d = c is an ellipse centred at μ, with axes along the eigenvectors uᵢ and half-lengths c × sqrt(λᵢ) (Bishop 2.3; Penn State STAT 505, 4.6). In the story the eigenvalues are 99.2 and 13.8, so the square roots are about 9.96 and 3.72, and the ellipse is 9.958 / 3.720 = 2.7 times longer than wide. The long axis points along (0.642, 0.767): height and weight rising together.

Step 8. Count the probability inside. In 1-D, within ±1 standard deviation sits 68 percent of the probability. In 2-D, the region d ≤ 1 holds 1 − e^(−1/2) = 0.3935, about 39 percent, because d² is chi-square with 2 degrees of freedom. More dimensions means a smaller share inside 1σ, so the "68-95-99.7" rule does not carry over unchanged.

Worked example 1: two classmates, same straight-line distance

Take mean μ = (170, 65) and Σ = [[49, 42], [42, 64]].

Determinant: det Σ = 49 × 64 − 42 × 42 = 3136 − 1764 = 1372.
Inverse of a 2 × 2 matrix: swap the diagonal, negate the off-diagonal, divide by the determinant. Σ⁻¹ = (1/1372) × [[64, −42], [−42, 49]] ≈ [[0.04665, −0.03061], [−0.03061, 0.03571]].
Classmate A is (177, 72). The difference is (7, 7). Euclidean distance: sqrt(49 + 49) = 9.899.
Mahalanobis d² for A: (1/1372) × (64 × 7² + 49 × 7² − 2 × 42 × 7 × 7) = 49 × (64 + 49 − 84) / 1372 = 49 × 29 / 1372 = 1.0357. So d = 1.018.
Classmate B is (177, 58). The difference is (7, −7). Euclidean distance is again 9.899.
Mahalanobis d² for B: 49 × (64 + 49 + 84) / 1372 = 49 × 197 / 1372 = 7.0357. So d = 2.652.
Interpretation: the covariance 42 is positive, so a +7 cm and +7 kg move agrees with the cloud's tilt (cheap in Mahalanobis terms), while +7 cm with −7 kg goes against the tilt (expensive). The same distance in a straight line is 1.02 versus 2.65 in the cloud's own units.
Density ratio: the factor exp(−½ d²) is exp(−0.518) = 0.596 for A and exp(−3.518) = 0.0297 for B, so A is about 20 times as likely (in density) as B.

Worked example 2: the density value at the mean and the ellipse axes

At the mean d = 0, so the exponential is 1 and the density is the normalizing constant alone: 1 / sqrt((2π)² × 1372) = 1 / (2π × sqrt(1372)) = 1 / (2π × 37.04) = 0.004297 per (cm × kg).
For A, the density is 0.004297 × 0.596 = 0.00196. For B it is 0.004297 × 0.0297 = 0.000127. (Densities are not probabilities; they need to be multiplied by an area.)
Eigenvalues: solve det(Σ − λI) = 0, so λ² − (49 + 64)λ + 1372 = 0, so λ² − 113λ + 1372 = 0. Then λ = (113 ± sqrt(12769 − 5488)) / 2 = (113 ± 85.33) / 2, giving 99.16 and 13.84. Check: the sum is 113 (the trace), and the product is 1372 (the determinant).
Standard deviations along the axes: sqrt(99.16) = 9.958 and sqrt(13.84) = 3.720. Ratio 2.677.
Eigenvectors: (0.642, 0.767) for 99.16 and (−0.767, 0.642) for 13.84. They are perpendicular, as expected for a symmetric matrix.
The 1σ ellipse (d = 1) therefore has half-lengths 9.96 along (0.642, 0.767) and 3.72 along (−0.767, 0.642), and it contains 39.35 percent of the probability.
Using chi-square with 2 degrees of freedom, a 95 percent ellipse has d² = −2 ln(0.05) = 5.99, so d = 2.45, and the 50 percent ellipse has d² = 1.386. Classmate B has d² = 7.04 > 5.99, so B lies outside the 95 percent ellipse, while A (d² = 1.04) is comfortably inside it.

Comparison tables

Quantity	Symbol	Story value	What it tells you
Mean vector	μ	(170, 65)	Centre of the cloud
Variances	Σ₁₁, Σ₂₂	49 and 64	Spread of each variable alone
Covariance	Σ₁₂	42	Joint movement, with units
Correlation	ρ	0.75	Joint movement, unit-free
Determinant	det Σ	1372	Squared "volume" of the cloud
Eigenvalues	λ	99.16 and 13.84	Variance along each ellipse axis
Peak density	p(μ)	0.004297	Normalizing constant

Covariance shape	Form of Σ	Contours	Independent variables?	Free parameters in n dimensions
Isotropic	σ²I	Circles or spheres	Yes	1
Diagonal	diag(σ₁², …, σₙ²)	Axis-aligned ellipses	Yes	n
Full	any symmetric PSD matrix	Tilted ellipses	Not in general	n(n+1)/2

Distance	Formula	Uses Σ?	Story, classmate A	Story, classmate B
Euclidean	sqrt of the sum of squared differences	No	9.90	9.90
Standardized (per variable)	each difference divided by its σ	Diagonal only	1.33	1.33
Mahalanobis	sqrt((x − μ)ᵀ Σ⁻¹ (x − μ))	Full	1.02	2.65

Dimension n	Share inside 1σ (d ≤ 1)	Note
1	0.6827	Familiar 68 percent
2	0.3935	1 − e^(−1/2)

Standardized per-variable distance is sqrt((7/7)² + (7/8)²) = 1.33 for both A and B: it fixes the units but cannot see the tilt, so it cannot tell them apart.

Everything the video skipped

Why Σ is always positive semidefinite. For any fixed vector v, the number vᵀx is a single random variable (a weighted sum of the components). Its variance is Var(vᵀx) = vᵀΣv. Variance can never be negative, so vᵀΣv ≥ 0 for every v. That is exactly the PSD condition. If vᵀΣv could be zero for some non-zero v, the data would lie exactly on a flat slice (for example one variable is a perfect copy of another), Σ would be singular, and the usual density formula would break because Σ⁻¹ does not exist.

Marginals and conditionals are Gaussian. If (x₁, x₂) is jointly Gaussian, then the marginal of x₁ (ignoring x₂) is Gaussian with the matching block of μ and Σ. The conditional of x₁ given a fixed x₂ is also Gaussian (MML 6.5.1; Bishop 2.3.1 and 2.3.2). For the story, the marginal of height is just N(170, 49), read off the diagonal. The conditional has a mean that shifts with the observed value and a smaller variance: for a 2-D Gaussian the conditional variance of weight given height is 64 − 42²/49 = 64 − 36 = 28, which is less than 64 because knowing the height removes part of the uncertainty. The conditional mean of weight given height h is 65 + (42/49)(h − 170), so each extra centimetre adds 0.857 kg on average.

Linear transformations. If x is Gaussian with mean μ and covariance Σ, then y = Ax + b is Gaussian with mean Aμ + b and covariance AΣAᵀ (MML 6.5.3). This is why whitening and rotations keep a Gaussian a Gaussian.

Sampling with Cholesky. To draw samples from N(μ, Σ), compute the Cholesky factor L with Σ = LLᵀ, draw z of independent standard normals, and return x = μ + Lz (MML 6.5.4). The covariance of Lz is L I Lᵀ = Σ. For the story, L = [[7, 0], [6, 5.2915]]: the first output is 7 z₁ (a height with standard deviation 7), and the second is 6 z₁ + 5.2915 z₂, which shares the height noise (that is the covariance 7 × 6 = 42) and adds its own independent part. The check: 6² + 5.2915² = 36 + 28 = 64.

Maximum likelihood, a teaser for 1.37. If you observe N independent samples from a Gaussian, the parameters that make the data most likely are the sample mean for μ and the sample covariance (dividing by N, which is the maximum likelihood version) for Σ (Bishop 2.3.4). So "fitting a Gaussian" is just computing averages. The next lesson, 1.37, explains why.

1-D versus 2-D coverage. The "68-95-99.7 rule" is a one-dimensional fact. In n dimensions the coverage of the d ≤ k region comes from the chi-square distribution with n degrees of freedom, so it shrinks as n grows for a fixed k.

Parameter count and the practical use of diagonal covariance. A full covariance matrix has n(n+1)/2 free numbers (n variances plus n(n−1)/2 distinct covariances), plus n for the mean. For n = 2 that is 3 for Σ; for n = 100 it is 5,050; for n = 1000 it is 500,500. Estimating this many numbers needs a lot of data, and inverting Σ costs time on the order of n³. Therefore practitioners often use diagonal or isotropic covariances, which have n or 1 parameters, when data are limited or the dimension is large, accepting that they cannot represent tilt. This is a common modelling choice rather than a rule.

Connections to other lessons.

1.35 Expectation rules and covariance: Σ is the table of every pairwise covariance from that lesson.
1.21 Basis change: the eigenvector matrix U is a change of basis into the cloud's own axes, where the covariance is diagonal.
The eigenvalue lessons in Module 1: the ellipse axes and the variances along them are Σ's eigenvectors and eigenvalues.
1.37 Maximum likelihood estimation: the next lesson, which fits μ and Σ from data.
Later modules (PCA, Gaussian mixture models, VAEs): PCA uses Σ's eigenvectors, mixture models use several Gaussians together, and VAEs sample from Gaussians.

Common mistakes and myths

Myth: zero covariance means independence. It does not in general (GBC 3.8; MML 6.4.5). The exception is the jointly Gaussian case, where zero covariance does imply independence.
Mistake: reading the covariance 42 as "strong". Covariance depends on units. Use the correlation (0.75) to judge strength.
Mistake: using Euclidean distance on mixed units. A centimetre and a kilogram are not comparable, and the straight line ignores the tilt.
Mistake: expecting 68 percent inside 1σ in 2-D. It is 39 percent. The 1-D rule does not carry over.
Mistake: confusing Σ with Σ⁻¹. The Mahalanobis distance uses the inverse (precision matrix), not Σ itself.
Mistake: forgetting the determinant factor. Without 1 / sqrt((2π)ⁿ det Σ) the function does not integrate to 1.
Mistake: treating density as probability. The density 0.004297 at the mean is not a probability; it is per unit area.
Myth: a Gaussian describes any cloud. Correlation measures only linear relationships; skewed or multi-modal data need other models (D2L 22.6.1.10 shows Y = X² with zero covariance).
Mistake: mixing up the two sample-covariance divisors. Dividing by N gives the maximum likelihood estimate; dividing by N − 1 gives the unbiased one.

Real-life uses

These are standard background applications.

Anomaly detection: fit a Gaussian to normal data and flag points whose Mahalanobis distance is large, as with classmate B.
PCA: the main directions of variation are Σ's eigenvectors, ranked by eigenvalue.
Gaussian mixture models: a cloud with several clusters is modelled as a weighted sum of Gaussians, each with its own mean and Σ.
Kalman filters: the uncertainty of a moving object's position is tracked as a covariance matrix that grows and shrinks as measurements arrive.
VAEs: the model samples a latent vector from a Gaussian and decodes it, usually with a diagonal covariance.
Finance and engineering: covariance matrices describe how several assets or sensors move together.

Self-check questions

What are the diagonal and off-diagonal entries of Σ? The diagonal holds variances; the off-diagonal entries hold covariances.
Compute the correlation from Σ = [[49, 42], [42, 64]]. 42 / (7 × 8) = 0.75.
Why is Σ symmetric? Cov(X, Y) = Cov(Y, X).
Why is Σ positive semidefinite? vᵀΣv = Var(vᵀx) ≥ 0 for every v.
Write the multivariate Gaussian density in words. One over the square root of ((2π)ⁿ times det Σ), times e to the minus one half of the squared Mahalanobis distance.
What does the determinant do in the density? It sets the normalizing constant so the total probability is 1; a bigger cloud gets a lower peak.
Two points have the same Euclidean distance from the mean. Can their Mahalanobis distances differ? Yes. In the story 9.9 gives 1.02 for (177, 72) and 2.65 for (177, 58).
What do the eigenvectors and eigenvalues of Σ give? The ellipse's axis directions and the variances along them; the half-lengths are proportional to the square roots of the eigenvalues (about 9.96 and 3.72).
How much of a 2-D Gaussian lies inside the 1σ ellipse? 1 − e^(−1/2) = 39 percent.
How many free parameters does a full covariance matrix have in n dimensions, and what is the alternative? n(n+1)/2; diagonal (n) or isotropic (1) covariances are common cheaper choices.

Key takeaways

Σ holds every variance and every covariance of the cloud.
Correlation is covariance rescaled to be unit-free.
The Mahalanobis distance measures distance in the cloud's own shape, using Σ⁻¹.
The density is 1 / sqrt((2π)ⁿ det Σ) × exp(−½ d²), and its contours are ellipses along Σ's eigenvectors.
In 2-D the 1σ ellipse holds 39 percent, not 68.
Marginals, conditionals and linear transformations of Gaussians are Gaussian.

Go deeper

Compute the conditional distribution of weight given height by hand, using the formulas in Bishop 2.3.1.
Draw samples with the Cholesky factor and check that the sample covariance approaches Σ.
Look at how whitening changes the ellipse into a circle.

Next roadmap ID: 1.37 Maximum likelihood estimation.

Sources and further reading

Deisenroth, Faisal, Ong, Mathematics for Machine Learning, ch. 6 "Probability and Distributions", sections 6.4.1 Means and Covariances, 6.4.5 Statistical Independence, 6.5 Gaussian Distribution (6.5.1 marginals and conditionals, 6.5.3 linear transformations, 6.5.4 sampling via Cholesky). https://mml-book.github.io/book/mml-book.pdf
Bishop, Pattern Recognition and Machine Learning, sec. 2.3 "The Gaussian Distribution" (Mahalanobis distance, eigenvectors and ellipsoids, 2.3.1 conditional, 2.3.2 marginal, 2.3.4 maximum likelihood). https://www.microsoft.com/en-us/research/wp-content/uploads/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf
Goodfellow, Bengio, Courville, Deep Learning, ch. 3 "Probability and Information Theory", sections 3.8 Expectation, Variance and Covariance and 3.9.3 Gaussian Distribution (precision matrix, diagonal and isotropic covariance). https://www.deeplearningbook.org/contents/prob.html
Zhang, Lipton, Li, Smola, Dive into Deep Learning, sec. 2.6 "Probability and Statistics" (2.6.6 Expectations, variance, covariance matrix). https://d2l.ai/chapter_preliminaries/probability.html
Zhang, Lipton, Li, Smola, Dive into Deep Learning, sec. 22.6 "Random Variables" (22.6.1.10 Covariance). https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/random-variables.html
Penn State STAT 505, 4.6 "Geometry of the Multivariate Normal Distribution" (prediction ellipse, axes along eigenvectors, half-lengths sqrt(λ χ²)). https://online.stat.psu.edu/stat505/book/export/html/659

Accuracy note: the height and weight data are illustrative; all numbers were checked in Python, and background applications are standard knowledge rather than claims from the sources above.

Search This Blog

Learn AI by Story

The covariance matrix and the multivariate Gaussian: study notes