1.22 Orthogonality and projections: study notes

Learn AI by Story

1. The idea in one sentence

To find the best approximation of a vector inside a line or a plane, drop a perpendicular onto it: the foot of that perpendicular (the "shadow") is the closest point, and the leftover gap is at a right angle (orthogonal) to everything you projected onto.

2. The story from the video

Asha is flying a kite at noon. The sun is straight overhead, so the kite string casts a shadow on the ground. Asha notices that the shadow is the part of the string that points along the ground, while the missing part (the height) points straight up, at a right angle to the ground. The shadow is the closest spot on the ground to the kite's tip, and any other spot on the ground is farther away. That shadow is a projection, and the straight-up gap is the part that is orthogonal to the ground.

3. Key definitions (glossary)

Term	Precise definition	Plain meaning
Vector	An ordered list of numbers, such as x = (1, 3), seen as an arrow or a point.	An arrow with a direction and a length.
Inner product (dot product)	For vectors in Rⁿ: xᵀy = x₁y₁ + x₂y₂ + ... + xₙyₙ. In general, a function of two vectors that is symmetric, linear in each slot, and positive for any nonzero vector with itself.	Multiply matching entries and add. It measures how much two arrows agree in direction.
Norm	The length of a vector: ‖x‖ = √(xᵀx). For (3, 4) it is 5.	How long the arrow is (see 1.17 Vector norms).
Angle	cos θ = xᵀy / (‖x‖ · ‖y‖), with θ between 0 and 180 degrees.	How far apart two arrows point; the dot product turns it into a number.
Orthogonal	Two vectors are orthogonal when xᵀy = 0.	They meet at a right angle (the zero vector counts as orthogonal to everything).
Orthonormal	Orthogonal and every vector has length 1.	Perpendicular and unit length, like the x-axis and y-axis arrows.
Orthonormal basis	A basis whose vectors are orthonormal.	A tidy set of perpendicular rulers that can describe every vector.
Orthogonal matrix	A square matrix Q whose columns are orthonormal, so QᵀQ = QQᵀ = I and Q⁻¹ = Qᵀ.	A pure rotation or reflection: it keeps lengths and angles.
Subspace	A set of vectors closed under adding and scaling (a line or plane through the origin, and so on).	The flat place you are projecting onto.
Orthogonal complement	For a subspace U, the set of all vectors orthogonal to every vector in U, written U⊥.	Everything that sticks out at a right angle from U.
Projection	A linear map π with π(π(x)) = π(x) (applying it twice changes nothing). The orthogonal projection also leaves a gap that is orthogonal to the subspace.	Casting a shadow onto a line or plane.
Projection matrix	A matrix P with π(x) = Px. For a line spanned by b: P = bbᵀ / bᵀb.	A machine that turns any vector into its shadow.
Idempotent	A matrix with P² = P.	Doing it again does nothing: a shadow of a shadow is the same shadow.
Residual (gap, error vector)	x − Px, the part of x that the projection misses.	The leftover piece, perpendicular to the line.
Gram-Schmidt	A procedure that turns any set of independent vectors into an orthonormal set: subtract the projections onto earlier vectors, then rescale.	A recipe for straightening slanted rulers into perpendicular ones.
QR decomposition	Writing A = QR with Q having orthonormal columns and R upper triangular. It is Gram-Schmidt in matrix form.	The stored result of straightening the columns of A.
Pseudo-inverse	For B with independent columns: (BᵀB)⁻¹Bᵀ. Used to find the best-fit coordinates.	A stand-in for an inverse when an exact one does not exist.
Full rank	A matrix whose columns (or rows) are all independent.	No column is redundant.
Jitter / ridge	Adding a small εI to BᵀB before inverting, for numerical stability.	A tiny nudge that keeps the arithmetic safe.

4. Detailed step-by-step explanation

Step 1: Orthogonal means dot product zero

Take b = (2, 1) and c = (−1, 2). The dot product is 2·(−1) + 1·2 = −2 + 2 = 0. So they are orthogonal. Sketch them and you will see c is b turned a quarter turn. The test "xᵀy = 0" works in any number of dimensions, even when you cannot draw the picture, and that is why it is so useful in machine learning, where vectors have hundreds or thousands of entries.

Why does the dot product detect right angles? From the cosine formula, cos θ = xᵀy / (||x|| ||y||). If the top is 0, then cos θ = 0, so θ = 90 degrees.

Step 2: Why perpendicular means closest

Pick a line through the origin in the direction b. You are standing at a point x off the line and you want the closest point on the line. Imagine walking along the line: the distance first shrinks and then grows again. It is smallest exactly where your straight path to the line meets it at a right angle. This is Pythagoras: for any other point q on the line, the squared distance is (gap)² + (distance from the foot to q)², which is bigger than the gap alone. So the foot of the perpendicular wins.

Step 3: Compute the shadow on a line

We want the foot to be some multiple of b, say λb. The gap x − λb must be orthogonal to b:

bᵀ(x − λb) = 0, so bᵀx = λ bᵀb, so λ = bᵀx / bᵀb.

The shadow is π(x) = λb. In words: measure how much of x points along b (the top bᵀx), divide by how long b is squared (the bottom), and scale b by that amount.

Step 4: Package it as a matrix

Because λ = bᵀx / bᵀb, the shadow is b(bᵀx)/bᵀb = (bbᵀ / bᵀb) x. So P = bbᵀ / bᵀb. The top bbᵀ is an outer product (a column times a row, giving a matrix), and the bottom bᵀb is just a number. P is symmetric (Pᵀ = P) and idempotent (P² = P).

Step 5: Why P² = P

Once a vector sits on the line, its shadow is itself, so projecting again changes nothing. This is the definition of a projection in the book: applying it twice equals applying it once.

Step 6: Going beyond a line

For a subspace spanned by the columns of a matrix B, the same idea (gap orthogonal to every column) leads to the normal equation BᵀBλ = Bᵀx, so λ = (BᵀB)⁻¹Bᵀx and the shadow is Bλ. This needs the columns of B to be independent (full rank). The projection matrix is B(BᵀB)⁻¹Bᵀ. If the columns are orthonormal, BᵀB = I and everything collapses to the simple form BBᵀ. That is why orthonormal bases are so loved: no inversion needed.

5. Everything the video skipped

Orthogonal matrices and rotations. If Q has orthonormal columns and is square, QᵀQ = I, so its inverse is just its transpose. Goodfellow's chapter 2 notes this is very cheap to compute. Such a matrix preserves lengths and angles, so it rotates or reflects without stretching. An example is Q = [[0.6, −0.8], [0.8, 0.6]].

At most n orthogonal directions in Rⁿ. In the plane you can have at most 2 mutually orthogonal nonzero vectors, in 3D at most 3, and so on.

Orthogonal is not the same as independent. Orthogonal nonzero vectors are always independent, but independent vectors need not be orthogonal. Gram-Schmidt fixes that by straightening them.

Gram-Schmidt in simple words. Keep the first vector (rescaled to length 1). For the second, subtract its shadow on the first, which leaves a piece that is orthogonal to the first, then rescale. Repeat for the third, subtracting shadows on all earlier ones. In practice a more stable variant (modified Gram-Schmidt, or Householder reflections) is used inside QR.

Orthogonal complement and the split x = shadow + gap. Every vector splits into one piece in the subspace U and one piece in U⊥, and the split is unique. The dimension of U plus the dimension of U⊥ equals n.

Angle between vectors. For (1, 1) and (1, 2) the dot product is 3, the lengths are √2 and √5, so cos θ = 3/√10 and θ is about 18.4 degrees (the book gives about 18 degrees).

Projection is not reversible. P squashes the whole plane onto a line. Many different x give the same shadow, so you cannot recover x from Px. P is not invertible (unless P = I).

Where the story connects in the roadmap.

1.17 Norms: the length of the gap, ||x − Px||, is the error measured by the Euclidean norm. The shadow is the point that minimises that norm.
1.20 Rank: P = bbᵀ/bᵀb has rank 1 (it maps onto a line). A projection onto a k-dimensional subspace has rank k.
1.21 Basis: an orthonormal basis is the nicest basis because coordinates are plain dot products.
1.22 and 1.23: least squares is exactly projection of the target vector onto the column space of the data matrix. The next video is the same picture with data.
Later: PCA projects data onto the directions of greatest spread (the book credits Pearson 1901 and Hotelling 1933 for the idea). Linear regression in Module 2 reuses the normal equation.

Limitations.

Orthogonality in the video is the usual dot-product kind. With a different inner product (a weighted one), "perpendicular" changes shape, and so does the closest point.
Projection finds the closest point in the Euclidean sense. If your error measure is different (for example absolute error), the answer changes.
With nearly dependent columns, (BᵀB)⁻¹ is numerically fragile. Practitioners use QR or the SVD, or add a small ridge term, instead of inverting directly.

6. Worked examples

Example A (from the video): shadow of x = (1, 3) on b = (2, 1)

bᵀx = 2·1 + 1·3 = 5.
bᵀb = 2·2 + 1·1 = 5.
λ = 5/5 = 1.
Shadow = 1·(2, 1) = (2, 1).
Gap = x − shadow = (1, 3) − (2, 1) = (−1, 2).
Check orthogonality: gap · b = (−1)(2) + (2)(1) = 0. Good.
Length of the gap = √(1 + 4) = √5, about 2.236.
Compare with another point on the line, (4, 2): the vector from x to it is (−3, 1), with length √10, about 3.162. That is larger, so the shadow really is closest.

Projection matrix for the same b. bbᵀ = [[4, 2], [2, 1]]; divide by 5: P = [[0.8, 0.4], [0.4, 0.2]]. Then P(1, 3) = (0.8 + 1.2, 0.4 + 0.6) = (2, 1), matching the shadow. P(2, 1) = (1.6 + 0.4, 0.8 + 0.2) = (2, 1), so a point already on the line is unchanged, and P² = P.

Rotation check. Q = [[0.6, −0.8], [0.8, 0.6]] has columns (0.6, 0.8) and (−0.8, 0.6). Each has length √(0.36 + 0.64) = 1 and their dot product is −0.48 + 0.48 = 0, so QᵀQ = I.

Example B (new): x = (4, 3) on b = (1, 2)

bᵀx = 1·4 + 2·3 = 10.
bᵀb = 1 + 4 = 5.
λ = 10/5 = 2.
Shadow = 2·(1, 2) = (2, 4).
Gap = (4, 3) − (2, 4) = (2, −1).
Check: gap · b = 2·1 + (−1)·2 = 0. Good.
Gap length = √(4 + 1) = √5, about 2.236.
Projection matrix: bbᵀ/5 = [[0.2, 0.4], [0.4, 0.8]]. Applying it to (4, 3) gives (0.8 + 1.2, 1.6 + 2.4) = (2, 4). Squaring the matrix returns the same matrix, so it is idempotent.

Example C (new): Gram-Schmidt on a₁ = (1, 1), a₂ = (1, 0)

Length of a₁ = √2, so u₁ = (1, 1)/√2, about (0.7071, 0.7071).
Shadow of a₂ on u₁: u₁ᵀa₂ = 0.7071, so shadow = 0.7071 · (0.7071, 0.7071) = (0.5, 0.5).
Subtract: v = a₂ − shadow = (0.5, −0.5). Check: v · a₁ = 0.5 − 0.5 = 0.
Rescale: length of v is √0.5, about 0.7071, so u₂ = (0.7071, −0.7071).
Now u₁ and u₂ are an orthonormal basis for the plane, built from two slanted vectors.

7. Comparison tables

	Independent	Orthogonal	Orthonormal
Meaning	None is a combination of the others	Pairwise dot product 0	Orthogonal and all lengths 1
Example	(1, 0), (1, 1)	(2, 1), (−1, 2)	(0.6, 0.8), (−0.8, 0.6)
Good for	Describing a space	Easy coordinates	Coordinates by plain dot products, Q⁻¹ = Qᵀ

	Projection onto a line	Projection onto a subspace
Formula for coordinates	λ = bᵀx / bᵀb	λ = (BᵀB)⁻¹Bᵀx
Projection matrix	bbᵀ / bᵀb	B(BᵀB)⁻¹Bᵀ
Needs inversion?	Only divides by a number	Inverts a small matrix (unless B is orthonormal)
Orthonormal case	bbᵀ when ‖b‖ = 1 (no division needed)	BBᵀ when the columns of B are orthonormal (BᵀB = I)

8. Common mistakes and myths

Mistake: forgetting to divide by bᵀb. The shadow is (bᵀx)b / bᵀb, not (bᵀx)b. It only works without dividing when b has length 1.
Myth: orthogonal means unrelated. It means no shared direction under the dot product. It does not mean the two things are unrelated in the real world.
Mistake: thinking orthogonal and independent are the same. Orthogonal implies independent for nonzero vectors, but not the other way around.
Mistake: projecting onto a vector that does not pass through the origin. Subspaces always include the origin.
Myth: projection can be undone. It throws information away.
Mistake: checking orthogonality by comparing numbers that look "opposite". (2, 1) and (−2, −1) are not orthogonal; their dot product is −5.
Mistake: with approximate numbers, expecting exactly 0. On a computer the dot product may come out as something like 1e-16, which is zero for practical purposes.

9. Real-life and AI uses

Least squares and regression: fitting a line is projecting the data vector onto the span of the feature columns (video 1.23).
PCA: projecting data onto a few orthogonal directions that keep the most variation.
Embeddings: a dot product near 0 between two embedding vectors means they share no direction, so similarity scores are low.
Compression and denoising: keep the shadow on the important directions and drop the rest.
Stable computing: orthogonal matrices (QR, SVD) do not blow up rounding errors; many neural-network initialisations and orthogonal-weight methods use this idea.
Graphics and physics: shadows, force components along a ramp, and so on.

10. Self-check questions

How do you test whether two vectors are orthogonal? Answer: their dot product is 0.
Are (2, 1) and (−1, 2) orthogonal? Answer: yes, 2·(−1) + 1·2 = 0.
What is the formula for the shadow of x on a line along b? Answer: (bᵀx / bᵀb) b.
Project (1, 3) on (2, 1). What do you get? Answer: (2, 1).
What is the gap in that example and how long is it? Answer: (−1, 2), length √5, about 2.236.
What does P² = P mean in plain words? Answer: projecting twice equals projecting once.
Why is the shadow the closest point? Answer: any other point adds a sideways leg to a right triangle, and Pythagoras makes the hypotenuse longer.
For an orthogonal matrix Q, what is Q⁻¹? Answer: Qᵀ.
In Gram-Schmidt, what do you subtract from the second vector? Answer: its shadow on the first (normalised) vector.
Can you recover x from its projection Px? Answer: no, projection loses the gap.

11. Key takeaways, go deeper, next

Key takeaways

Orthogonal means dot product 0.
Shadow on a line: (bᵀx / bᵀb) b; matrix bbᵀ / bᵀb.
The gap is orthogonal to the line and its length is the smallest possible distance.
P² = P; projection is lossy.
Orthonormal bases make everything simple (Q⁻¹ = Qᵀ).

Go deeper

Read the book sections on angles, orthogonality and orthogonal projections (see sources).
Try Gram-Schmidt on three vectors in R³ with Python.
Verify numerically that P² = P and Pᵀ = P for a random b.
Think about how a projection onto a plane changes the formula.

Next roadmap ID: 1.23 Least squares and the normal equation.

12. Sources and further reading

Accuracy note: The vectors in this document are small illustrative numbers chosen so the arithmetic is clean; all were re-checked with Python. The kite story is a made-up teaching picture, and the PCA dates come from the Mathematics for Machine Learning text.

Search This Blog