Expectation rules and covariance: study notes

 

1.35 Expectation rules and covariance: study notes

Learn AI by Story

The idea in one sentence

An expectation is a probability-weighted average that passes straight through sums and scaling, variance measures spread (and scales with the square), and covariance measures whether two quantities move together, with correlation as its scale-free version.

The story from the video

Priya's ice-cream cart sells 40, 60 or 80 cones a day with probabilities 0.25, 0.5 and 0.25, so she expects 60 cones. Each cone earns 20 rupees and rent is 300, so profit is 20X minus 300 and the expected profit is 900. The variance of cones is 200, so the profit variance is 400 times 200 = 80,000 (standard deviation about 283 rupees); the rent shifts the mean but not the spread. Then she records four days of temperature against cones: 20, 25, 30, 35 degrees and 40, 50, 70, 80 cones. The means are 27.5 and 60; the products of deviations average to a covariance of 87.5, and dividing by both standard deviations gives a correlation of 0.99. (All of Priya's data is illustrative.)

Key definitions (glossary)

Expectation (expected value, mean). For a discrete random variable, E[X] is the sum over x of x times p(x); for a continuous one it is the integral of x f(x) dx. Plain English: the long-run average, the balance point of the distribution. It need not be a value the variable can actually take.

Law of the unconscious statistician (LOTUS). To get E[g(X)] you do not need the distribution of g(X); just compute the sum over x of g(x) p(x), or the integral of g(x) f(x) dx. Plain English: apply the function to each value, then weight by the original probabilities. Note that E[g(X)] is generally not g(E[X]).

Linearity of expectation. E[aX + b] = a E[X] + b, and E[X + Y] = E[X] + E[Y] for any random variables, dependent or not (Deisenroth et al., 6.4.4). Plain English: averages of sums are sums of averages, with no conditions at all. This is the single most useful fact in the chapter.

Variance. Var(X) = E[(X minus E[X])^2]: the average squared distance from the mean. Plain English: how spread out the values are, in squared units.

Standard deviation. The square root of the variance, in the original units. Plain English: the typical distance from the mean.

The shortcut E[X^2] minus E[X]^2. Expanding the square gives Var(X) = E[X^2] − (E[X])^2. Plain English: the mean of the squares minus the square of the mean. It is easier to compute, and it shows variance is never negative.

Variance rules. Var(aX + b) = a^2 Var(X): shifting does nothing, scaling multiplies the variance by the square of the factor. For independent X and Y, Var(X + Y) = Var(X) + Var(Y); in general Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y) (D2L 22.6.1.5; Deisenroth et al., 6.4.4).

Covariance. Cov(X, Y) = E[(X − E[X])(Y − E[Y])] = E[XY] − E[X] E[Y]. Plain English: the average product of the two deviations. Positive means they tend to be above their means together; negative means one is above when the other is below; zero means no linear tendency. It measures only linear relationships (D2L 22.6.1.10).

Correlation. Corr(X, Y) = Cov(X, Y) / (sd(X) sd(Y)), always between −1 and +1. Plain English: covariance with the units stripped out, so it is comparable across problems. Goodfellow et al. (3.8) note that it normalises the scale.

Sample versus population covariance (n versus n−1). The population (or "plug-in") covariance divides the sum of deviation products by n. The sample covariance divides by n−1 (Bessel's correction), which makes it an unbiased estimate of the true covariance when the means were themselves estimated from the same data. Plain English: use n when the data is the whole population you care about, n−1 when the data is a sample from a bigger one. Deisenroth et al. cover the empirical versions in 6.4.2.

Chebyshev's inequality. For any random variable with mean mu and standard deviation sigma, P(abs(X − mu) at least k sigma) is at most 1/k^2, for k greater than 0. Plain English: no matter the shape, few values can be far from the mean in units of standard deviation (D2L 22.6.1.6).

Uncorrelated versus independent. Independent implies uncorrelated, because E[XY] = E[X] E[Y]. Uncorrelated (Cov = 0) does not imply independent: covariance sees only linear association. Plain English: zero covariance rules out a straight-line trend, not every kind of relationship (Goodfellow et al., 3.8; Deisenroth et al., 6.4.5).

Random variable, pmf and pdf. See 1.34. A pmf gives probabilities at points; a pdf gives density, and expectations become integrals.

Step-by-step explanation

Step 1. Expected cones. Multiply each value by its probability and add: 40 times 0.25 = 10, 60 times 0.5 = 30, 80 times 0.25 = 20. Total E[X] = 60 cones.

Step 2. Expected profit by linearity. Profit is 20X − 300, so E[profit] = 20 times 60 − 300 = 900 rupees. No need to list profits for each outcome (though you can: 500, 900, 1,300 with the same probabilities gives 125 + 450 + 325 = 900).

Step 3. Variance of cones. Deviations from 60 are −20, 0, +20; squares 400, 0, 400; weighted by 0.25, 0.5, 0.25 gives 100 + 0 + 100 = 200. Standard deviation is the square root of 200, which is 14.14 cones.

Step 4. Variance of profit. Var(20X − 300) = 20^2 times 200 = 400 times 200 = 80,000. The standard deviation is 282.84 rupees (which equals 20 times 14.14). Subtracting 300 moves the mean but changes no distances.

Step 5. Four days of data. Temperatures 20, 25, 30, 35 have mean 27.5; cones 40, 50, 70, 80 have mean 60. Deviations in temperature: −7.5, −2.5, +2.5, +7.5. Deviations in cones: −20, −10, +10, +20.

Step 6. Products of deviations. (−7.5)(−20) = 150; (−2.5)(−10) = 25; (2.5)(10) = 25; (7.5)(20) = 150. Sum 350.

Step 7. Covariance. Population: 350 / 4 = 87.5. Sample: 350 / 3 = 116.67. Both are positive: hot days and high sales go together.

Step 8. Standard deviations. Temperature: squared deviations 56.25, 6.25, 6.25, 56.25 sum 125, divided by 4 gives 31.25, square root 5.590. Cones: 400, 100, 100, 400 sum 1,000, divided by 4 gives 250, square root 15.811.

Step 9. Correlation. 87.5 / (5.590 times 15.811) = 87.5 / 88.39 = 0.98995, about 0.99. The sample versions give the same number, because the n or n−1 factor cancels between numerator and denominator.

Step 10. Variance of a sum. Var(T + C) = Var(T) + Var(C) + 2 Cov(T, C) = 31.25 + 250 + 2 times 87.5 = 456.25. The extra 175 is the covariance contribution; if the two were independent it would be absent.

Everything the video skipped

Continuous versions. For a continuous X with density f, E[X] is the integral of x f(x) dx, E[g(X)] is the integral of g(x) f(x) dx, and Var(X) is the integral of (x − mu)^2 f(x) dx. Covariance uses the joint density f(x, y) and a double integral. All the algebraic rules (linearity, the shortcut, variance of a sum) are identical; only sums become integrals (roadmap ID 1.27).

E[XY] = E[X] E[Y] for independent variables. If X and Y are independent, the joint factorises, so the double sum or integral of x y p(x) p(y) splits into the product of the two single expectations. Then Cov = E[XY] − E[X] E[Y] = 0. This is why independent implies uncorrelated, and why the variance of an independent sum has no cross term.

Zero covariance without independence: the Y = X squared example. Let X take the values −1, 0, 1 with probability 1/3 each, and Y = X^2. Then E[X] = 0, E[XY] = E[X^3] = (−1 + 0 + 1)/3 = 0, so Cov(X, Y) = 0 − 0 times E[Y] = 0. Yet Y is completely determined by X. Covariance cannot see a symmetric U-shaped relationship (D2L 22.6.1.10 uses a similar example; Deisenroth et al., Example 6.5).

Correlation as the cosine of an angle. Treat centred random variables as vectors with inner product Cov(X, Y). The length of a vector is its standard deviation, so correlation is the inner product divided by the product of lengths: the cosine of the angle between them. Correlation +1 means the same direction, 0 means perpendicular (uncorrelated is orthogonal), −1 means opposite (Deisenroth et al., 6.4.6). This explains the bound of −1 to +1 (Cauchy–Schwarz).

Correlation measures only linear association. A perfect curved relationship can have a small correlation. Also, correlation is not causation: temperature and cones are correlated, but an ice-cream cart that sells more does not warm the day.

Chebyshev's inequality in action. For Priya's cones the standard deviation is 14.14. The deviation of 20 is k = 20 / 14.14 = 1.414 standard deviations, so Chebyshev bounds P(abs(X − 60) at least 20) by 1/k^2 = 0.5. The true value is P(X = 40 or X = 80) = 0.5. This distribution meets the bound exactly, so Chebyshev cannot be improved in general.

Variance of sums with several variables. For n variables the variance of the sum is the sum of all variances plus twice the sum of all pairwise covariances. This is why averaging many models helps most when their errors do not co-vary: the covariance terms are what stop the variance of the average from shrinking.

Roadmap connections. The next video, roadmap ID 1.36, stacks all variances and covariances into the covariance matrix and builds the multivariate Gaussian. PCA in later modules starts from the covariance matrix of the features and finds its eigenvectors. Roadmap ID 1.38 Bayesian updating reuses expectations over posteriors, and ensembles, portfolio theory and gradient noise (variance of a stochastic gradient) all use the variance-of-a-sum formula.

Worked example 1: Priya's profit, three ways

1. Direct. Profit values 20 times 40 − 300 = 500, 20 times 60 − 300 = 900, 20 times 80 − 300 = 1,300, with probabilities 0.25, 0.5, 0.25. E = 125 + 450 + 325 = 900.
2. By linearity. 20 times E[X] − 300 = 20 times 60 − 300 = 900. Same answer.
3. Variance directly. Deviations from 900 are −400, 0, +400; squares 160,000, 0, 160,000; weighted: 40,000 + 0 + 40,000 = 80,000.
4. Variance by the rule. 20^2 times 200 = 80,000. Same answer, standard deviation 282.84.
5. Variance by the shortcut. E[X^2] = 0.25 times 1,600 + 0.5 times 3,600 + 0.25 times 6,400 = 400 + 1,800 + 1,600 = 3,800. Minus E[X]^2 = 3,600 gives 200. Matches.
6. Chebyshev check. The standard deviation of profit is 282.84; the profit moves 400 from the mean, which is k = 1.414 standard deviations, and the bound 1/k^2 = 0.5 is again met exactly by P(profit is 500 or 1,300) = 0.5.

Worked example 2: temperature, cones and the variance of a sum

1. Means: temperature 27.5, cones 60. Deviation products 150, 25, 25, 150 (sum 350).
2. Population covariance 350 / 4 = 87.5; sample covariance 350 / 3 = 116.67.
3. Standard deviations (population) 5.590 and 15.811, correlation 87.5 / 88.39 = 0.99.
4. Check the E[XY] − E[X]E[Y] formula. Products T times C: 20 times 40 = 800, 25 times 50 = 1,250, 30 times 70 = 2,100, 35 times 80 = 2,800. Their average is 6,950 / 4 = 1,737.5. Subtract 27.5 times 60 = 1,650, leaving 87.5. Same covariance by the shortcut.
5. Variance of the sum directly: daily sums T + C are 60, 75, 100, 115 with mean 87.5; deviations −27.5, −12.5, +12.5, +27.5; squares 756.25, 156.25, 156.25, 756.25 sum 1,825; divided by 4 gives 456.25. Matches 31.25 + 250 + 2 times 87.5 = 456.25.
6. Variance of the difference: 31.25 + 250 − 175 = 106.25. Positive covariance makes the sum more variable and the difference less variable.

Comparison tables

Quantity	Formula	Units	Priya value
Mean of cones	sum of x times p(x)	cones	60
Variance of cones	E[(X − mean)^2]	cones squared	200
Standard deviation	square root of variance	cones	14.14
Covariance of temperature and cones	average product of deviations	degrees times cones	87.5 (population)
Correlation	covariance over both standard deviations	none	0.99

Convention	Divide by	Temperature and cones covariance	When to use
Population (plug-in)	n = 4	87.5	The data is everything you care about
Sample (Bessel)	n − 1 = 3	116.67	The data is a sample from a larger population

Transformation	Mean	Variance
X plus b	E[X] + b	Var(X), unchanged
aX	a E[X]	a squared times Var(X)
X plus Y (any)	E[X] + E[Y]	Var X + Var Y + 2 Cov(X, Y)
X plus Y (independent)	E[X] + E[Y]	Var X + Var Y

Property	Independent	Uncorrelated
Definition	Joint equals product of marginals	Covariance is zero
Implies the other?	Implies uncorrelated	Does not imply independent
Detects	Any kind of relationship	Only linear relationships
Example of the gap	Y = X squared with symmetric X is dependent	Same example has Cov = 0

Common mistakes and myths

• Believing expectation of a function equals the function of the expectation. E[X^2] is 3,800 for the cones, but E[X]^2 is 3,600. The gap is exactly the variance.
• Thinking linearity needs independence. E[X + Y] = E[X] + E[Y] always. Only the variance rule for sums and E[XY] factorisation need independence (or zero covariance).
• Forgetting to square the scale. Var(20X) is 400 times Var(X), not 20 times.
• Thinking a shift changes variance. Subtracting rent changed the mean by 300 and the spread by nothing.
• Myth: zero covariance means independent. The Y = X squared example refutes it.
• Myth: correlation shows causation. It measures association only, and a hidden third variable (here, weather) can drive both.
• Mixing n and n−1. Quote which one you used. The covariance changes from 87.5 to 116.67, but the correlation does not change.
• Comparing covariances across different units. Covariance in degrees times cones has no universal scale; use correlation.
• Reading a correlation of 0.99 from four points as proof. Four days is too few to be confident; the correlation is a description of this sample.

Real-life uses

• PCA starts with the covariance matrix of the features and keeps the directions of largest variance.
• Ensembles and model averaging reduce variance most when members' errors have low covariance.
• Portfolio risk is the variance of a weighted sum of returns, which includes all the pairwise covariance terms.
• Stochastic gradient descent has noise whose variance shrinks as the batch size grows, using the variance of an average.
• Feature selection and data cleaning use correlation to spot redundant features.
• Forecasting and planning (Priya's cart, stock levels, staffing) use expectation for the plan and standard deviation for the safety margin.

Self-check questions

1. A fair six-sided die: what is E[X]? Answer: (1 + 2 + 3 + 4 + 5 + 6) / 6 = 3.5.
2. What is E[2X + 5] for that die? Answer: 2 times 3.5 + 5 = 12.
3. Variance of 3X + 7 if Var(X) = 4? Answer: 9 times 4 = 36.
4. Use the shortcut: if E[X] = 3 and E[X^2] = 13, what is Var(X)? Answer: 13 − 9 = 4.
5. Two variables have Var = 9 and Var = 16 and covariance 6. Variance of the sum? Answer: 9 + 16 + 12 = 37.
6. Same variables: correlation? Answer: 6 / (3 times 4) = 0.5.
7. Why can correlation never exceed 1 in absolute value? Answer: it is the cosine of an angle between centred variables (Cauchy–Schwarz).
8. Give an example of zero covariance with dependence. Answer: X symmetric around 0 and Y = X squared.
9. When do you divide by n−1? Answer: when estimating covariance from a sample, to correct the bias from using the sample mean.
10. What does Chebyshev say about a value 3 standard deviations from the mean? Answer: at most 1/9, about 11 percent, of the probability can be that far out, for any distribution.

Key takeaways

• Expectations add up, always; linearity needs no independence.
• Variance squares the scale and ignores shifts.
• Covariance asks whether two things move together; correlation rescales it to between −1 and +1.
• Zero covariance is not independence, and the variance of a sum includes twice the covariance.

Go deeper

• Compute the covariance of your own two-column dataset by hand using both conventions.
• Prove E[X^2] − E[X]^2 equals the variance by expanding the square.
• Show that Corr between X and aX + b is +1 for positive a and −1 for negative a.
• Try a symmetric example other than Y = X squared (for instance Y = abs(X)) and check the covariance.

Next on the roadmap: 1.36 The covariance matrix and the multivariate Gaussian.

Sources and further reading

• Zhang, Lipton, Li, Smola, Dive into Deep Learning, section 22.6 "Random Variables" (22.6.1.4 Means, 22.6.1.5 Variances, 22.6.1.6 Standard deviations and Chebyshev, 22.6.1.10 Covariance). https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/random-variables.html
• Zhang et al., Dive into Deep Learning, section 2.6 "Probability and Statistics" (2.6.6 Expectations, variance, covariance matrix). https://d2l.ai/chapter_preliminaries/probability.html
• Goodfellow, Bengio, Courville, Deep Learning (2016), chapter 3, section 3.8 "Expectation, Variance and Covariance". https://www.deeplearningbook.org/contents/prob.html
• Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapter 6: 6.4.1 Means and Covariances, 6.4.2 Empirical Means and Covariances, 6.4.4 Sums and Transformations of Random Variables, 6.4.5 Statistical Independence, 6.4.6 Inner Products of Random Variables. https://mml-book.github.io/book/mml-book.pdf

Accuracy note: Priya's cone probabilities, temperatures and sales are illustrative; all arithmetic was checked exactly.