1.15 Eigenvectors and SVD: study notes

Learn AI by Story

Concept 1.15, the directions that matter. Video story key: p115. Previous: 1.14 Cross-entropy and KL divergence. Next: 1.16 Landscapes: convex, local minima, saddle points.

1. The idea in one sentence

An eigenvector of a matrix is a direction the matrix only stretches (never turns), and the singular value decomposition (SVD) extends this to every matrix of any shape by writing it as rotate, stretch, rotate, so the biggest stretches show which directions matter most.

2. The story from the video

Pia presses on a rubber sheet and watches little arrows move. Most arrows turn, but a few special ones only get longer or shorter. With the matrix A = [[2, 1], [2, 3]], the arrow (1, 1) becomes (3, 5), a different direction, while (1, 2) becomes (4, 8), exactly 4 times itself. So (1, 2) is an eigenvector with eigenvalue 4; the other eigenvalue is 1 with eigenvector (1, −1). The video then moves to SVD, A = UΣVᵀ, using B = [[4, 0], [3, −5]] whose singular values are √40 ≈ 6.32 and √10 ≈ 3.16, so the first stretch holds 40 out of 50, or 80 percent, of the matrix's squared size. It ends with PCA, compression and recommenders.

3. Key definitions

Vector. A list of numbers, drawn as an arrow. Plain English: a direction with a length.
Matrix. A grid of numbers that acts on vectors, turning an input vector into an output vector (matrix times vector). Plain English: a machine that stretches, squashes, rotates and mixes arrows.
Linear transformation. What a matrix does: straight lines stay straight and the origin stays put.
Eigenvector. A nonzero vector v with Av = λv. The matrix changes only its length (and possibly flips it), never moves it off its own line. Plain English: a direction the matrix leaves alone.
Eigenvalue λ. The stretch factor for an eigenvector. A negative λ flips the arrow; a fraction squashes it.
Identity matrix I. The matrix that changes nothing; ones on the diagonal, zeros elsewhere.
Determinant det(A). A single number summarising how much the matrix scales area (or volume). For a 2×2 matrix [[a, b], [c, d]] it is ad − bc. It equals the product of the eigenvalues.
Trace. The sum of the diagonal entries; it equals the sum of the eigenvalues.
Characteristic equation. det(A − λI) = 0; its roots are the eigenvalues.
Symmetric matrix. A matrix equal to its transpose (A = Aᵀ). Plain English: a mirror image across the diagonal.
Transpose Aᵀ. The matrix flipped across its diagonal (rows become columns).
Orthogonal matrix. A matrix whose columns are mutually perpendicular unit vectors, so QᵀQ = I. It rotates or reflects without stretching.
Eigendecomposition. Writing A = WΛW⁻¹ with eigenvectors as columns of W. It exists only when there is a full set of independent eigenvectors.
Singular value decomposition (SVD). A = UΣVᵀ for any real m×n matrix, with U and V orthogonal and Σ diagonal with nonnegative entries.
Singular values σ. The diagonal entries of Σ, sorted largest first. They are the stretch factors along special perpendicular input directions.
Left and right singular vectors. The columns of U and V. Right singular vectors are eigenvectors of AᵀA; left ones are eigenvectors of AAᵀ; and σ² is an eigenvalue of AᵀA.
Rank. The number of independent directions a matrix actually uses; the number of nonzero singular values.
Low-rank approximation. Keeping only the largest few singular values to get a simpler matrix close to the original.
Covariance matrix and PCA. The covariance matrix records how features vary together (see 1.11). Principal component analysis (PCA) uses its top eigenvectors as the directions of greatest spread.

4. Detailed step-by-step explanation

Step 1: A matrix as an arrow-mover

Multiplying a matrix by a vector produces a new vector. For A = [[2, 1], [2, 3]] and input (x, y), the output is (2x + y, 2x + 3y). Try (1, 1): (2+1, 2+3) = (3, 5). The input pointed at 45 degrees; the output points steeper. Most inputs behave like this: they turn.

Step 2: Spotting an arrow that does not turn

Try (1, 2): (2×1 + 1×2, 2×1 + 3×2) = (4, 8) = 4 × (1, 2). Same direction, four times as long. This is an eigenvector with eigenvalue 4. Try (1, −1): (2 − 1, 2 − 3) = (1, −1) = 1 × (1, −1). It is also an eigenvector, with eigenvalue 1 (no change).

Step 3: Why such directions exist, and how to find them

We want Av = λv for a nonzero v. Rewrite it as (A − λI)v = 0. A nonzero v can only be sent to zero if the matrix A − λI squashes some direction flat, which happens exactly when its determinant is zero: det(A − λI) = 0. For our A this gives (2 − λ)(3 − λ) − 2 = λ² − 5λ + 4 = 0, so λ = 1 or 4. Substituting each λ back into (A − λI)v = 0 gives the eigenvectors. This procedure is the one given in the d2l.ai and 3Blue1Brown pages.

Av = \lambda v \;\Longleftrightarrow\; (A - \lambda I)v = 0 \;\Longrightarrow\; \det(A - \lambda I) = 0

Step 4: Why the biggest eigenvalue matters

If you apply A again and again to almost any starting arrow, the component along the largest-eigenvalue direction grows fastest and takes over. This is power iteration. The d2l.ai page shows it with random matrices: repeated multiplication makes the vector's length grow geometrically, at a rate settling to the largest-magnitude eigenvalue, while the vector aligns with the principal eigenvector. So the biggest eigenvalue marks the direction a matrix amplifies most.

Step 5: The special case of symmetric matrices

When A equals its transpose, the eigenvalues are real and the eigenvectors can be chosen perpendicular (orthogonal). Then A = QΛQᵀ with Q orthogonal: rotate into the eigen-directions, stretch each by its eigenvalue, rotate back. Covariance matrices are symmetric, which is why PCA is clean.

Step 6: When eigenvectors fall short

Eigenvectors need a square matrix, and not every square matrix has enough of them. The shear [[1, 1], [0, 1]] has only one eigen-direction. A 90-degree rotation of the plane has no real eigenvectors at all (3Blue1Brown). Non-square matrices (say 1000 users by 50 movies) have no eigenvectors by definition. We want something that always works.

Step 7: SVD for any matrix

SVD says every real matrix A can be written A = UΣVᵀ. Read it right to left on an input vector: Vᵀ rotates (or reflects) the input, Σ stretches along the axes by σ1 ≥ σ2 ≥ … ≥ 0, and U rotates the result into the output space. MIT 18.065 (Prof. Gilbert Strang) describes it as splitting a matrix into rank-one pieces ordered by importance: A = σ1 u1 v1ᵀ + σ2 u2 v2ᵀ + …

A = U \Sigma V^{T} = \sigma_1 u_1 v_1^{T} + \sigma_2 u_2 v_2^{T} + \cdots

Step 8: How SVD connects to eigenvectors

The right singular vectors are the eigenvectors of AᵀA, the left singular vectors are the eigenvectors of AAᵀ, and each σ² is a (nonzero) eigenvalue of both (MIT 18.065; Deep Learning book chapter 2). That is how you compute SVD by hand: find eigenvalues of AᵀA, take square roots.

Step 9: The video's matrix

B = [[4, 0], [3, −5]]. Then BᵀB = [[25, −15], [−15, 25]], whose eigenvalues are 40 and 10 (trace 50, determinant 625 − 225 = 400 = 40 × 10). So σ1 = √40 = 6.3246 and σ2 = √10 = 3.1623. Share of squared size: 40 / (40 + 10) = 80 percent. The first piece alone, σ1 u1 v1ᵀ, equals [[2, −2], [4, −4]], the best rank-1 approximation, and the leftover error has size σ2 = 3.162 (a known result called the Eckart-Young theorem; named here only).

Step 10: Keep the big pieces

Because pieces are ordered by importance, dropping the smallest singular values gives the best simplification of a given rank. That one idea powers compression, noise removal, PCA and recommenders.

5. Everything the video skipped

Eigendecomposition in full. If A has n independent eigenvectors, put them in the columns of W and the eigenvalues on the diagonal of Λ: A = WΛW⁻¹. Powers become easy: Aᵏ = WΛᵏW⁻¹. For our A, the eigenvalues of A² are 1 and 16. 3Blue1Brown describes this as changing to an "eigenbasis" in which the matrix is diagonal.

Useful identities. The determinant is the product of the eigenvalues and the trace is their sum. For A = [[2, 1], [2, 3]], det = 6 − 2 = 4 = 1 × 4 and trace = 5 = 1 + 4. Zero eigenvalues mean the matrix flattens some direction and is singular (not invertible).

Negative and complex eigenvalues. A negative eigenvalue flips the arrow; the line is still preserved. Rotations have complex eigenvalues, so no real eigenvector exists.

Pseudoinverse. From the SVD, A⁺ = V D⁺ Uᵀ (D⁺: invert the nonzero singular values). Per the Deep Learning book it gives the minimum-norm solution of underdetermined systems and the closest-fit solution of overdetermined ones, connecting SVD to least squares.

PCA. Per the Deep Learning book, PCA's best directions are the top eigenvectors of XᵀX, where X stacks centred data points as rows, which is the same thing as the right singular vectors of X.

Initialisation and scaling. The d2l.ai page links the largest eigenvalue to neural-network weight initialisation: if a weight matrix has largest eigenvalue much above 1, repeated layers can blow signals up; much below 1, they shrink them.

Variants. Truncated SVD keeps the top k singular values. Compact SVD drops zero ones. Randomised SVD speeds this up for huge matrices (named as pointers only). Generalised eigenproblems and non-symmetric complex cases exist but are beyond this concept.

Limitations. Eigenvectors need square matrices and may be missing. SVD finds linear structure only; twisted, curved data is poorly summarised. Large singular value means large variance or energy, not necessarily importance for a task. Computing a full SVD on a very large matrix is expensive.

Roadmap links. Earlier: 1.07 chain rule, 1.08 probability basics, 1.11 mean, variance and correlation (covariance and correlation feed PCA), 1.14 cross-entropy. Later: 1.16 landscapes, where Hessian eigenvalues tell minima from saddle points, and 1.17 vector norms, which measure the lengths that singular values stretch.

6. Worked examples

All arithmetic below was checked in Python (numpy).

Worked example A: eigenvalues of the video's matrix, from scratch

A = [[2, 1], [2, 3]].

Form A − λI = [[2 − λ, 1], [2, 3 − λ]].
Determinant: (2 − λ)(3 − λ) − 1×2 = 6 − 5λ + λ² − 2 = λ² − 5λ + 4.
Factor: (λ − 1)(λ − 4) = 0, so λ = 1 or λ = 4.
For λ = 4: (A − 4I) = [[−2, 1], [2, −1]]. The first row says −2x + y = 0, so y = 2x: eigenvector (1, 2).
For λ = 1: (A − I) = [[1, 1], [2, 2]]. The first row says x + y = 0, so y = −x: eigenvector (1, −1).
Checks: A(1, 2) = (4, 8) = 4(1, 2); A(1, −1) = (1, −1). Trace 5 = 1 + 4; determinant 4 = 1 × 4.
Numpy returns unit-length versions: (0.447, 0.894) and (−0.707, 0.707) (a different length and sign are equally valid).

Worked example B: power iteration finds the big direction

Start with (1, 0) and repeatedly multiply by A (rescaling only to read off the direction):

Step	Vector	Direction as unit vector
0	(1, 0)	(1.000, 0.000)
1	(2, 2)	(0.707, 0.707)
2	(6, 10)	(0.514, 0.857)
3	(22, 42)	(0.464, 0.886)
4	(86, 170)	(0.451, 0.892)
5	(342, 682)	(0.448, 0.894)

The direction converges to (1, 2) / √5 = (0.447, 0.894), the eigenvector for the largest eigenvalue 4. The vector also grows by a factor close to 4 per step (682 / 170 = 4.01), which is the eigenvalue showing itself.

Worked example C: SVD of the video's matrix B

B = [[4, 0], [3, −5]].

BᵀB = [[4×4 + 3×3, 0 + 3×(−5)], [−15, 25]] = [[25, −15], [−15, 25]].
Its characteristic equation: λ² − 50λ + 400 = 0, so λ = 40 and λ = 10.
Singular values: σ1 = √40 = 6.3246, σ2 = √10 = 3.1623.
Right singular vectors: for 40, (1, −1)/√2; for 10, (1, 1)/√2.
Check: B(1, −1)/√2 = (4, 8)/√2 = (2.83, 5.66), of length √(8 + 32) = 6.3246 = σ1, pointing along (1, 2): so u1 = (1, 2)/√5.
Squared-size share: 40 / (40 + 10) = 0.80, which also equals 80 percent of the sum of the squares of all entries (16 + 0 + 9 + 25 = 50).
Best rank-1 approximation: σ1 u1 v1ᵀ = [[2, −2], [4, −4]]. Its difference from B is [[2, 2], [−1, −1]], whose size is √(4 + 4 + 1 + 1) = √10 = 3.162 = σ2.

Worked example D: a PCA direction (illustrative covariance)

Suppose two features have covariance matrix C = [[4, 2], [2, 3]] (illustrative). It is symmetric.

Trace 7 and determinant 12 − 4 = 8, so λ² − 7λ + 8 = 0 and λ = (7 ± √17) / 2 = 5.562 and 1.438.
The top eigenvector is about (0.788, 0.615); the other, (−0.615, 0.788), is perpendicular to it.
The top direction holds 5.562 / 7 = 79.5 percent of the total variance. Keeping only that one coordinate would retain about four fifths of the spread.

Worked example E: a low-rank matrix and storage (illustrative)

R = [[5, 5, 0], [4, 4, 0], [0, 0, 3]] has singular values 9.055, 3.000 and about 0 (to rounding), so its rank is 2. Two rank-one pieces rebuild it exactly. For a 1000 × 1000 matrix, 1,000,000 numbers are stored fully; a rank-1 approximation needs only 1000 + 1000 + 1 = 2,001 numbers (illustrative sizes).

7. Comparison tables and diagrams

Feature	Eigendecomposition	SVD
Applies to	Square matrices with a full eigenvector set	Every real matrix, any shape
Form	A = WΛW⁻¹ (symmetric: QΛQᵀ)	A = UΣVᵀ
Special numbers	Eigenvalues (may be negative or complex)	Singular values (always zero or positive)
Directions	One set of eigenvectors (orthogonal if symmetric)	Two orthogonal sets, U and V
Meaning	Directions left unturned	Rotate, stretch, rotate
Link	Aᵏ is easy	Eigenvectors of AᵀA and AAᵀ

Matrix	Eigenvalues	Eigenvectors	Note
[[2, 1], [2, 3]]	1, 4	(1, −1), (1, 2)	Video example
[[1, 1], [0, 1]]	1, 1	one only: (1, 0)	Shear, no full set
90-degree rotation	complex only	none real	Turns every arrow
[[3, 0], [0, 1]]	3, 1	the axes	Diagonal: singular values 3, 1

Diagram of SVD acting on an arrow:

input x --V^T--> rotated --Sigma--> stretched --U--> output Ax
         turn       axes get longer/shorter       turn again

Singular value	Squared	Share of total
σ1 = 6.325	40	80%
σ2 = 3.162	10	20%

8. Common mistakes and myths

Myth: every matrix has eigenvectors you can use. Non-square matrices have none, and some square ones lack a full set (the shear).
Mistake: forgetting the zero vector is excluded. Av = λv is trivially true for v = 0, so eigenvectors must be nonzero.
Myth: an eigenvector has a unique length. Any nonzero multiple is also an eigenvector; libraries return unit length and the sign may differ.
Mistake: mixing up singular values and eigenvalues. They coincide for symmetric positive matrices only. Generally σ² is an eigenvalue of AᵀA, not of A.
Myth: eigenvalues of a sum are the sums of eigenvalues. This is false in general; the identities that hold are trace = sum and determinant = product.
Mistake: reading the percentage as accuracy. The 80 percent in the video is the squared-size share of one stretch, not a prediction score.
Myth: the biggest singular value is always the most useful signal. It captures the most variance or energy, which may be noise or an uninteresting trend.
Mistake: forgetting to centre data before PCA. PCA directions are about spread around the mean, so subtract the mean first (see 1.11).

9. Real-life uses with concrete examples

PCA. Finds the directions of most spread in data, for example reducing hundreds of survey answers to two or three summary axes that can be plotted.
Image compression. A grey picture is a matrix of pixel values; keeping the top k singular values stores about k × (rows + columns + 1) numbers instead of rows × columns, losing only fine detail.
Recommender systems. A users-by-items rating matrix is approximated with a few hidden taste factors; each user and item gets a short vector of those factors.
Least squares and pseudoinverse. SVD gives a stable way to solve over- or under-determined linear systems.
Neural-network scaling. Largest-eigenvalue (and largest singular value) behaviour of weight matrices explains why signals grow or shrink through many layers, linked to initialisation (d2l.ai).
Noise removal. Dropping the smallest singular values often removes noise while keeping structure.
Dynamics and stability. Repeatedly applying a matrix, as in simulation, is governed by its eigenvalues: those above 1 in size grow.

10. Self-check questions

State the eigenvector equation. Av = λv with v nonzero.
How do you find eigenvalues? Solve det(A − λI) = 0.
For A = [[2, 1], [2, 3]], what is A(1, 2)? (4, 8) = 4 × (1, 2), so (1, 2) is an eigenvector with λ = 4.
What are the eigenvalues of A and what do trace and determinant tell you? 1 and 4; trace 5 = sum, determinant 4 = product.
Why is (1, 1) not an eigenvector of A? A(1, 1) = (3, 5), which is not a multiple of (1, 1).
Why is SVD more general than eigendecomposition? It exists for every real matrix of any shape, not just square ones with enough eigenvectors.
State the SVD and say what each factor does. A = UΣVᵀ: Vᵀ rotates, Σ stretches, U rotates.
How are singular values related to eigenvalues? σ² equals an eigenvalue of AᵀA (and AAᵀ).
For B in the video, why is the share 80 percent? σ1² = 40 and σ2² = 10, so 40 / 50 = 0.8.
What is the size of the error after the best rank-1 approximation? It equals the dropped singular value σ2 = 3.162 (Eckart-Young, known result).

11. Key takeaways, go deeper, and what is next

Key takeaways

An eigenvector keeps its direction under a matrix; the eigenvalue is the stretch.
Eigenvalues come from det(A − λI) = 0; trace is their sum and determinant their product.
SVD works for any matrix: rotate, stretch, rotate, with singular values sorted by importance.
Keeping the biggest singular values gives the best low-rank approximation: the basis of PCA, compression and recommenders.
Symmetric matrices are the friendly case: real eigenvalues, perpendicular eigenvectors.

Go deeper

Work the 3Blue1Brown eigenvalue example by hand, then try MIT 18.065 problem 6 on [[3, 4], [0, 5]] (its singular values are about 6.708 and 2.236).
Run power iteration in a few lines of numpy.
Do PCA on a tiny two-feature dataset and plot the two directions.
Read about truncated and randomised SVD.

Next roadmap ID: 1.16 Landscapes: convex, local minima, saddle points.

12. Sources and further reading

Eigendecomposition, Dive into Deep Learning, d2l.ai: eigenvectors, the 2×2 example, symmetric matrices, power iteration.
Lecture 6: Singular Value Decomposition (SVD), MIT OpenCourseWare 18.065, Gilbert Strang, 2018: SVD as rank-one pieces, link to eigenvectors of AᵀA.
Linear Algebra, Deep Learning book chapter 2, Goodfellow, Bengio and Courville: eigendecomposition, SVD, pseudoinverse, PCA.
Eigenvectors and eigenvalues, 3Blue1Brown: visual explanation, characteristic polynomial, shear and rotation cases.
Information Theory appendix, d2l.ai: the previous concept (1.14), for context.

Accuracy note: The matrices A and B and their eigenvalues, singular values and the 80 percent share come from the video and were re-checked in numpy. The covariance matrix in example D, the matrix R and the 1000 × 1000 storage count in example E are illustrative numbers chosen for these notes. The Eckart-Young theorem, truncated and randomised SVD are named as well-known pointers; I did not open a source specific to them. Unit-length eigenvectors may differ in sign between libraries.

Search This Blog