1.11 Mean, variance, standard deviation and correlation: study notes

Learn AI by Story

1. The idea in one sentence

The mean tells you where a pile of numbers is centred, the variance and standard deviation tell you how spread out the numbers are around that centre, and correlation tells you whether two lists of numbers move together in a straight-line way.

2. The story from the video

Zoya takes five tests and scores 4, 8, 6, 5 and 7. Her friend asks, "So how good are you, and how steady are you?" Zoya adds the scores (30) and divides by five to get her average, 6. Then she measures how far each score sits from 6, squares those gaps so they cannot cancel, averages them (variance 2) and takes the square root (standard deviation about 1.41). Finally she wonders whether the hours she studies are linked to her scores, and a single number, the correlation, answers that.

3. Key definitions

Every term below has a precise meaning and a plain-English one-liner.

Term	Precise definition	Plain English
Data point	One recorded value, written x₁, x₂, … xₙ	One number in the list
n	The count of data points	How many numbers you have
Population	The complete set you care about	Everyone or everything, not just a few
Sample	A subset drawn from the population	A small handful you actually measured
Mean (average)	Sum of all values divided by n: x̄ = (x₁ + … + xₙ) ÷ n	The balance point of the numbers
Median	The middle value after sorting (or the average of the two middle values)	The number in the middle of the line
Mode	The value that occurs most often	The most popular number
Outlier	A value far away from the rest	The odd one out
Deviation (gap)	A value minus the mean: xᵢ − x̄	How far this number sits from the centre
Squared deviation	The gap multiplied by itself	The gap made positive and bigger when large
Population variance (σ²)	The mean of the squared deviations: Σ(xᵢ − μ)² ÷ n	Average squared distance from the mean
Sample variance (s²)	Sum of squared deviations divided by n − 1	Variance, adjusted because a sample under-estimates spread
Standard deviation (SD)	The square root of the variance	A typical distance from the mean, in the original units
Bessel's correction	Using n − 1 instead of n for a sample	A small fix so the sample estimate is not too low
Expected value E[x]	The probability-weighted average of x	The mean, seen from probability
z-score	(value − mean) ÷ SD	How many SDs above or below the mean
Covariance	The mean of (x gap) × (y gap) over all pairs	Do x and y sit above and below their means together?
Correlation (Pearson r)	Covariance ÷ (SD of x × SD of y), always between −1 and +1	Covariance with the units removed
Spearman correlation	Pearson r computed on the ranks instead of the raw values	Do they rise together in order, straight or curved?
Correlation matrix	A table of r for every pair of variables	A scoreboard of who moves with whom
Confounder	A hidden third variable that influences both variables	The secret cause behind a fake link
Standardisation	Subtracting the mean and dividing by the SD	Putting every feature on the same ruler
Line of best fit	The straight line y = a + bx that minimises squared vertical errors	The tidiest straight line through the dots

4. Step-by-step explanation

Step 1: Find the centre with the mean

Add every score and divide by how many there are. For Zoya: 4 + 8 + 6 + 5 + 7 = 30, and 30 ÷ 5 = 6. Why? Because we want a single number that represents the whole list. The mean is the balance point: if you put the scores on a see-saw, it balances at 6.

Step 2: Measure each gap from the mean

Subtract 6 from each score: −2, +2, 0, −1, +1. Why? Spread means "how far from the centre", so the gaps are the raw ingredient.

Step 3: Notice that simple adding fails

The gaps sum to −2 + 2 + 0 − 1 + 1 = 0. This is not a coincidence. The mean is defined as the balance point, so the gaps above and below always cancel exactly. Averaging the raw gaps would always give zero, which tells us nothing about spread.

Step 4: Square the gaps

Squared gaps: 4, 4, 0, 1, 1. Squaring makes every gap positive, so nothing cancels. It also punishes big gaps more than small ones (a gap of 4 counts sixteen times as much as a gap of 1). The sum is 10.

Step 5: Average the squares to get the variance

10 ÷ 5 = 2. This is the population variance. Its weakness is that it is in squared units: if the scores are marks, the variance is in "marks squared", which nobody can picture.

Step 6: Take the square root to get the standard deviation

√2 ≈ 1.41. Now we are back in marks. A typical score of Zoya's lands about 1.41 marks away from her mean of 6. Why the root? It undoes the squaring and returns to the original units.

Step 7: Compare spread between people

Leo scores 6, 6, 6, 6, 6: every gap is 0, so SD = 0. Mia scores 2, 10, 6, 3, 9, also with mean 6, but her gaps are −4, +4, 0, −3, +3, squared 16, 16, 0, 9, 9, which sum to 50. Variance 50 ÷ 5 = 10 and SD ≈ 3.16. All three have the same mean, yet very different consistency. The mean alone hides this.

Step 8: Move from one list to two lists

Now we have two lists: hours studied x = 1, 2, 3, 4, 5 and score y = 2, 4, 5, 4, 5. Means: x̄ = 3 and ȳ = 4. For each student we multiply the x gap by the y gap. If both are above their means (or both below) the product is positive; if one is above and the other below, it is negative.

Hours x	Score y	x gap	y gap	Product
1	2	−2	−2	+4
2	4	−1	0	0
3	5	0	+1	0
4	4	+1	0	0
5	5	+2	+1	+2
Sum				6

Step 9: Scale to correlation

Add up the squared x gaps (4 + 1 + 0 + 1 + 4 = 10) and the squared y gaps (4 + 0 + 1 + 0 + 1 = 6). Then r = 6 ÷ √(10 × 6) = 6 ÷ √60 ≈ 6 ÷ 7.746 ≈ 0.775. Dividing by the spreads is what squeezes the answer into the range −1 to +1, so that r does not depend on whether hours were measured in minutes or marks out of 10.

Step 10: Read the answer

r = +1 means the two rise together on a perfect straight line. r = 0 means no straight-line link. r = −1 means one rises exactly as the other falls. Here r ≈ 0.775 is a fairly strong positive link. The best-fit line is y = 2.2 + 0.6x: each extra hour goes with about 0.6 extra marks. The slope is r × (SD of y ÷ SD of x) = 0.7746 × (1.0954 ÷ 1.4142) = 0.6, and the line always passes through the point of means (3, 4): 2.2 + 0.6 × 3 = 4. ✔

Finally, remember that correlation is not causation. The numbers show a link, not a reason.

5. Everything the video skipped

Mean, median and mode, and outliers

The mean uses every number, so one extreme value drags it. Take 1, 2, 3, 4, 100: the mean is 22 but the median is 3. For incomes, house prices or website load times (which have a few huge values) the median usually describes a typical case better. The mode is the only one of the three that works for categories, such as the most common shirt size. For the symmetric bell curve from 1.10, mean, median and mode all land together.

Population versus sample variance

When your data is the whole population, divide by n. When your data is only a sample used to estimate a bigger population, divide by n − 1. The reason: a sample is centred on its own mean, which is, by construction, closer to the sample points than the true population mean is. So the squared gaps come out too small on average, and dividing by the slightly smaller n − 1 corrects that. This is Bessel's correction, named after the astronomer Friedrich Bessel (19th century).

Zoya's numbers, treated as a sample of her many tests: sample variance = 10 ÷ 4 = 2.5, and sample SD = √2.5 ≈ 1.58 (versus 1.41 when treated as a population). With a big n the difference vanishes: dividing by 999 or 1000 is nearly identical.

The shortcut formula

Variance = E[x²] − (E[x])², "the mean of the squares minus the square of the mean". For Zoya: squares 16, 64, 36, 25, 49 sum to 190, so E[x²] = 38; (E[x])² = 6² = 36; 38 − 36 = 2. ✔ Same answer, one pass through the data, which is why computers like it. A caution: with huge numbers it can lose precision, so careful software uses a stable alternative.

The 68-95-99.7 rule (link to 1.10)

For data shaped like a normal bell curve, about 68% of values lie within 1 SD of the mean, about 95% within 2 SDs and about 99.7% within 3 SDs. This is why SD is so useful: with just the mean and SD of a bell-shaped quantity you can sketch the whole distribution. The rule is only approximate for data that is not bell-shaped.

z-scores

A z-score is (value − mean) ÷ SD. Zoya's score of 8 gives z = (8 − 6) ÷ 1.41 ≈ 1.41, meaning it is 1.41 SDs above her usual. Her 4 gives z = −1.41. Z-scores let you compare things measured on different scales, such as a 72 in maths versus an 88 in history, by asking how unusual each is for its own subject. In practice, values beyond about ±3 are often flagged as possible outliers.

Covariance, and how it becomes correlation

Covariance is the average of the products of gaps. For the hours/score data the products sum to 6, so the population covariance is 6 ÷ 5 = 1.2 (sample covariance, dividing by 4, is 1.5). Its sign tells direction, but its size depends on units, so a covariance of 1.2 alone is hard to judge. Dividing by both SDs fixes this: r = 1.2 ÷ (1.414 × 1.095) = 1.2 ÷ 1.549 ≈ 0.775. So correlation is simply covariance scaled to the range −1 to +1. (Dividing by n or n − 1 in both numerator and denominator gives the same r.)

Pearson versus Spearman

Pearson r measures straight-line association using the actual values. Spearman's ρ first replaces values with their ranks (1st, 2nd, …) and then computes Pearson on the ranks. It captures any consistently rising or falling relationship, curved or not, and it is less sensitive to outliers. Example: x = 1, 2, 3, 4, 5 and y = 1, 4, 9, 16, 100. Spearman = 1.00 because y rises with x in perfect order, but Pearson is only about 0.80 because the relationship is not straight and the 100 dominates.

Correlation is not causation

Ice cream sales and drownings both rise in summer, so they correlate positively. Ice cream does not cause drowning: hot weather (the confounder) pushes both up. A correlation may mean A causes B, B causes A, a hidden third thing causes both, or pure coincidence. Only a designed experiment (for example a randomised trial) can establish cause with confidence.

Nonlinear relationships

Correlation measures only straight-line association. If x = −3, −2, −1, 0, 1, 2, 3 and y = x², the dots form a perfect U-shaped curve, yet r = 0 exactly, because the left side cancels the right. Always draw the scatter plot first.

Outliers distort r

Take x = 1, 2, 3, 4, 5 and y = 2, 3, 4, 5, 6: a perfect line, r = 1.0. Change the last y to −10 and r drops to about −0.57. One rogue point turned a perfect relationship into a negative one.

The correlation matrix

With many variables (say house size, rooms, age, price), we compute r for every pair and lay the results in a square table. The diagonal is always 1 (each variable with itself) and the table is symmetric. Data scientists scan it to spot features that duplicate each other.

Cosine similarity as uncentred correlation (link to 1.04)

Cosine similarity from 1.04 is the dot product of two vectors divided by their lengths. Pearson correlation is exactly the cosine similarity of the two lists after each has had its mean subtracted. So correlation is cosine similarity of centred data. On the raw hours and scores (without centring) the cosine is about 0.96, much higher than r = 0.775, because uncentred data all point in a similar positive direction.

History in one paragraph

Francis Galton studied how children's heights relate to their parents' in the 1880s and coined "regression" and an early idea of correlation. His student Karl Pearson formalised the product-moment correlation coefficient in the 1890s, which is why it is called Pearson's r. The term "standard deviation" was also introduced by Pearson (1893), and the word "variance" came from Ronald Fisher in 1918.

Connections across the roadmap

Earlier: 1.04 gave us dot products and cosine similarity, which hide inside correlation. 1.08 and 1.09 gave probability, which defines expected value. 1.10 gave the normal distribution whose two parameters are exactly a mean and an SD. Later: 1.12 (sampling and the law of large numbers) explains why sample means settle near the true mean as n grows, and 1.13 (entropy) measures a different kind of "spread", the uncertainty in a distribution.

6. Worked examples

Example A: a dataset with a clean answer

Data: 2, 4, 4, 4, 5, 5, 7, 9 (n = 8).

Mean: 2 + 4 + 4 + 4 + 5 + 5 + 7 + 9 = 40, and 40 ÷ 8 = 5.
Gaps from the mean: −3, −1, −1, −1, 0, 0, +2, +4. They sum to zero, a good check.
Squared gaps: 9, 1, 1, 1, 0, 0, 4, 16, which sum to 32.
Population variance: 32 ÷ 8 = 4.
Population SD: √4 = 2.
If this were a sample: variance 32 ÷ 7 ≈ 4.57 and SD ≈ 2.14.
Median: sorted already, middle two are 4 and 5, so median = 4.5. Mode = 4.
Shortcut check: squares 4, 16, 16, 16, 25, 25, 49, 81 sum to 232, so E[x²] = 29, and 29 − 5² = 4. ✔
z-score of 9: (9 − 5) ÷ 2 = +2, so 9 is two SDs above the mean.

Example B: a negative correlation

A (made-up, illustrative) café records the hours of afternoon sun on its terrace, x = 1, 2, 3, 4, 5, versus cups of hot chocolate sold, y = 9, 7, 6, 3, 2.

Means: x̄ = 15 ÷ 5 = 3 and ȳ = 27 ÷ 5 = 5.4.
x gaps: −2, −1, 0, +1, +2. y gaps: +3.6, +1.6, +0.6, −2.4, −3.4.
Products: −7.2, −1.6, 0, −2.4, −6.8, which sum to −18.
Σ(x gap)² = 4 + 1 + 0 + 1 + 4 = 10.
Σ(y gap)² = 12.96 + 2.56 + 0.36 + 5.76 + 11.56 = 33.2.
r = −18 ÷ √(10 × 33.2) = −18 ÷ √332 = −18 ÷ 18.22 ≈ −0.988.
Best-fit line: slope = −18 ÷ 10 = −1.8, intercept = 5.4 − (−1.8 × 3) = 10.8, so y = 10.8 − 1.8x.
Population covariance = −18 ÷ 5 = −3.6.

Reading it: r ≈ −0.99 is a very strong negative straight-line link; each extra hour of sun goes with about 1.8 fewer hot chocolates. Sun does plausibly cause this (warmer weather), but the numbers themselves cannot prove it.

Example C: covariance from scratch (hours and scores)

Using the table in Step 8: population covariance = 6 ÷ 5 = 1.2. SD of x = √(10 ÷ 5) = √2 ≈ 1.414. SD of y = √(6 ÷ 5) = √1.2 ≈ 1.095. r = 1.2 ÷ (1.414 × 1.095) = 1.2 ÷ 1.549 ≈ 0.775. The same answer as Step 9, confirming the two routes agree.

7. Comparison tables and diagrams

Measures of centre

Measure	How to get it	Strength	Weakness
Mean	Add and divide	Uses all data, has nice maths	Dragged by outliers
Median	Middle after sorting	Robust to outliers	Ignores how large the extreme values are
Mode	Most frequent	Works for categories	May be several, or none

Measures of spread

Measure	Formula in words	Units	Notes
Range	Largest minus smallest	Original	Uses only two numbers
Variance	Mean squared gap	Squared	Easy for maths, hard to picture
Standard deviation	Square root of variance	Original	The everyday choice
Sample versions	Divide by n − 1	Same	Use when estimating from a sample

The three people side by side

Person	Scores	Mean	Variance	SD
Leo	6, 6, 6, 6, 6	6	0	0
Zoya	4, 8, 6, 5, 7	6	2	1.41
Mia	2, 10, 6, 3, 9	6	10	3.16

Reading r

r (illustrative bands)	Meaning
+1	Perfect straight line, both rise together
about +0.7	Strong positive link
about +0.3	Weak positive link
0	No straight-line link (a curve is still possible)
about −0.7	Strong negative link
−1	Perfect straight line, one rises as the other falls

Text picture of the spread

Leo   :          6 6 6 6 6           (all stacked, SD 0)
Zoya  :        4 5 6 7 8             (SD 1.41)
Mia   :   2  3     6     9  10       (SD 3.16)
          1 2 3 4 5 6 7 8 9 10

8. Common mistakes and myths

Myth: "Correlation proves one thing causes the other." It only describes association. Confounders and coincidence are real.
Mistake: thinking r = 0 means "no relationship". It means no straight-line relationship. A U-shaped curve can give r = 0.
Mistake: adding raw gaps and wondering why the answer is always zero. Gaps cancel; that is why we square them.
Mistake: quoting variance as if it were in the original units. It is in squared units; use SD for a feel.
Mistake: using n for a small sample when you should use n − 1.
Mistake: trusting the mean when there are outliers. Check the median too.
Myth: "A high r means a steep line." Slope and r are different: r says how tightly dots hug a line, slope says how steep the line is.
Mistake: computing r without looking at a scatter plot. Outliers and curves can fool the number.
Myth: "A larger SD is always bad." It is just a description; for an investor it means risk, for a doctor it may mean real variety between patients.
Mistake: comparing covariances across different units. Use r instead.

9. Real-life and AI/ML uses

Everyday life: weather forecasts quote an average temperature but the SD tells you whether to pack a jacket; quality control in factories watches the SD of bolt sizes; investors measure risk as the SD of returns; schools use z-scores to compare exams.

Feature standardisation in machine learning. Imagine (illustrative) a model with two features: age (about 20 to 70) and yearly income (about 20,000 to 200,000). Methods that use distances or gradients, such as k-nearest neighbours, SVMs and neural networks trained with gradient descent, would be dominated by income simply because its numbers are bigger. Standardising each feature (subtract its mean, divide by its SD) gives both a mean of 0 and an SD of 1, so they contribute fairly and training converges faster. A key rule: compute the mean and SD on the training data only and reuse them for new data.

Batch normalisation. Inside a deep network, each layer's outputs can drift in scale as training changes earlier layers. Batch normalisation takes a mini-batch of activations, subtracts their mean, divides by their SD (with a tiny constant to avoid dividing by zero), and then applies a learned scale and shift. It is standardisation applied inside the network, and it usually makes training faster and more stable.

Bias-variance trade-off (brief). Here "variance" means how much a model's predictions would change if it were trained on different samples of data. A very flexible model has high variance (it chases noise, overfitting); a too-simple model has high bias (it misses real patterns). Good modelling balances the two.

Correlation matrices and feature selection. Before training, a data scientist checks which features correlate strongly with the target (useful) and which correlate strongly with each other (redundant, a problem called multicollinearity). Similarly, PCA (principal component analysis) uses the covariance matrix to find the directions of greatest variance.

Weight initialisation. Methods such as Xavier and He initialisation choose the SD of the starting weights so that signals neither shrink to zero nor blow up from layer to layer. This is variance bookkeeping.

Evaluating models. Reporting a model's accuracy as "mean ± SD over several runs" shows both how good and how stable it is. Mean squared error, the standard loss for regression, is literally a "mean of squared gaps" like the variance, just measured from predictions instead of from the mean.

10. Self-check questions

What is the mean of 3, 5, 7, 9, and what is the sum of the gaps from that mean? Answer: The sum is 24, so the mean is 6. The gaps are −3, −1, +1, +3, which sum to 0. Gaps from the mean always sum to zero.
Why do we square the gaps instead of just averaging them? Answer: Raw gaps always cancel to zero. Squaring makes them all positive, so the result measures spread, and it also weights large gaps more.
Zoya's variance is 2. What is her SD and what does it mean? Answer: √2 ≈ 1.41 marks. A typical score sits about 1.41 marks away from her mean of 6.
When do you divide by n − 1, and what are Zoya's sample variance and SD? Answer: When your data is a sample used to estimate a larger population. Sample variance = 10 ÷ 4 = 2.5 and SD ≈ 1.58.
For 1, 2, 3, 4, 100, which is more representative, the mean or the median? Answer: The median (3). The mean (22) is dragged up by the outlier 100.
A test has mean 70 and SD 10. What is the z-score of 90, and roughly what share of a bell-shaped class scores above it? Answer: z = (90 − 70) ÷ 10 = 2. About 95% lie within 2 SDs, so about 2.5% lie above +2 SDs.
If covariance is 1.2, SD of x is 1.414 and SD of y is 1.095, what is r? Answer: r = 1.2 ÷ (1.414 × 1.095) ≈ 0.775.
Can r be 0 when x and y are strongly related? Give an example. Answer: Yes. For y = x² with x from −3 to 3 the dots form a U-shape but r = 0, because correlation sees only straight-line patterns.
Ice cream sales and drownings correlate. Why is it wrong to say ice cream causes drowning? Answer: Hot weather is a confounder: it increases both. Correlation alone does not show cause.
Why do we standardise features before training many machine-learning models? Answer: To put features on the same scale (mean 0, SD 1) so large-valued features do not dominate distances or gradients, and so training is faster and steadier.

11. Key takeaways

Mean = the centre; variance and SD = the spread; correlation = the straight-line togetherness of two lists.
Gaps from the mean cancel, so we square them; the square root brings us back to original units.
Divide by n for a whole population and n − 1 for a sample. For Zoya, population SD ≈ 1.41, sample SD ≈ 1.58.
Correlation is covariance divided by both SDs, so it always lies between −1 and +1: hours and scores gave r ≈ 0.775, the café data gave r ≈ −0.988.
r = 0 does not mean no relationship, outliers can wreck r, and correlation is never proof of causation.
In machine learning, standardising features and batch normalisation both rely directly on the mean and SD.

Go deeper

Look up the median absolute deviation (MAD), a robust alternative to SD.
Read about the coefficient of determination R², which for a straight-line fit equals r squared (here 0.775² = 0.6, so the hours explain about 60% of the variation in scores).
Explore Spearman and Kendall rank correlations.
Learn about the covariance matrix and PCA.
Compute a correlation matrix for a small dataset yourself in a spreadsheet.
Read about Simpson's paradox, where a trend reverses when groups are combined.

Next roadmap ID: 1.12 Sampling and the law of large numbers

Learn AI by Story: one concept, one story, every day.

Mean, variance, standard deviation and correlation: study notes