1.21 Basis and change of basis: study notes

Learn AI by Story

1. The idea in one sentence

A basis is a set of arrows that you agree to count in, and a change of basis is a translation between two people's ways of counting: the vector itself never moves, only the numbers that describe it change.

2. The story from the video

Jen draws her own grid on the map. Her two arrows are tilted and stretched compared with ours: her "one step across" is the arrow (2, 1) and her "one step up" is the arrow (-1, 1). When Jen says "go -1 across and 2 up", we have to translate: that is -1 of her first arrow plus 2 of her second, which lands at (-4, 1) on our grid. The spot on the map is the same for both of us. Only the address changed. Going the other way, our address (3, 2) is, in Jen's language, (5/3, 1/3).

3. Key definitions

Vector: an arrow from the origin, or equivalently the point at its tip. Plain meaning: a movement or a location.
Linear combination: a sum of vectors, each multiplied by a number, such as 3a + 2b. Plain meaning: mix some arrows in chosen amounts.
Span: the set of all vectors reachable by linear combinations of given vectors (the Span lesson from 3Blue1Brown calls it "the set of all possible vectors you can reach with linear combinations of a given pair of vectors"). Plain meaning: everywhere your arrows can take you.
Linear independence: vectors are independent when each one adds a new direction to the span, so none is a combination of the others. Plain meaning: no arrow is redundant.
Basis: a set of linearly independent vectors that span the space. MIT 18.06SC puts the same idea as "a set of vectors, as few as possible, whose combinations produce all vectors in the space". Plain meaning: the smallest toolkit of arrows that can reach everywhere.
Dimension: the number of vectors in a basis. MIT 18.06SC states that this number equals the dimension of the space. Plain meaning: how many independent directions there are.
Standard basis: the default arrows i-hat = (1, 0) and j-hat = (0, 1) in the plane. Plain meaning: our usual grid.
Coordinates (relative to a basis): the amounts of each basis vector needed to build a vector. Plain meaning: the address in that grid's language.
Change-of-basis matrix M: the matrix whose columns are the new basis vectors written in standard coordinates. It converts new-basis coordinates into standard coordinates. Plain meaning: Jen-to-us translator.
Inverse matrix M^-1: the matrix that undoes M, so that M^-1 M = I. It converts standard coordinates into new-basis coordinates. Plain meaning: us-to-Jen translator.
Identity matrix I: the matrix that changes nothing. Plain meaning: "do nothing".
Determinant: the factor by which a matrix scales areas. Plain meaning: how much the grid cell grows or shrinks.
Similar matrices (conjugation): matrices A and M^-1 A M that describe the same motion in two different bases. Plain meaning: one motion, two descriptions.
Orthonormal basis: a basis whose vectors are perpendicular and have length 1. The video did not use this term; it is the nicest kind of basis and comes back in 1.22.
Eigenbasis: a basis made of eigenvectors of a matrix. Not in the video; it connects to 1.15.

4. Step-by-step explanation

Step 1. Coordinates are instructions, not locations. When we write (3, 2) we really mean "3 times i-hat plus 2 times j-hat". The numbers are scaling amounts. The 3Blue1Brown lesson makes the same point: the basis vectors are the things the coordinate numbers scale. Everyone who uses i-hat and j-hat agrees on what the numbers mean, so we forget there was a choice at all.

Step 2. Nothing forces that choice. Jen picks b1 = (2, 1) and b2 = (-1, 1). Her grid is tilted and the cells have a different size. Both grids share the same origin, because everyone agrees on what the zero vector is. Only the arrows and the spacing differ.

Step 3. What makes a valid basis. In the plane you need two vectors that are not parallel. If they line up, their span is only a line, and you cannot reach the rest of the plane. The Span lesson says that for most pairs of vectors you can reach every point; lined-up pairs are the exception. Jen's pair works: (2, 1) and (-1, 1) point in different directions.

Step 4. From Jen's language to ours. Jen's coordinates (-1, 2) mean -1 of b1 plus 2 of b2. In our numbers: -1 times (2, 1) is (-2, -1), and 2 times (-1, 1) is (-2, 2). Adding gives (-4, 1). We just did a matrix-vector product without naming it.

Step 5. Package it as a matrix. Put Jen's arrows as columns: M = [[2, -1], [1, 1]] (written by rows). Then M times her coordinates equals our coordinates. This is the same "columns are where the basis vectors land" idea from 1.18. M sends our i-hat to her first arrow and our j-hat to her second arrow. Read it as: "take what Jen means by (x, y) and give me the standard address".

Step 6. From our language to Jen's. We need the reverse trip, which is the inverse. The determinant of M is 2*1 - (-1)*1 = 3, which is not zero, so M has an inverse. For a 2x2 matrix, swap the diagonal entries, negate the off-diagonal entries, and divide by the determinant. That gives M^-1 = (1/3) [[1, 1], [-1, 2]]. The determinant also tells us something visual: one cell of Jen's grid has area 3 on our grid.

Step 7. Translating a motion. Suppose a motion is known in our coordinates, such as the 90 degree counterclockwise rotation R = [[0, -1], [1, 0]]. How would Jen write that same rotation as a matrix in her coordinates? Read the product M^-1 R M from right to left: first translate Jen's vector into ours (M), then rotate in our language (R), then translate the result back into hers (M^-1). The 3Blue1Brown lesson describes this three-step recipe.

Step 8. Why this matters. Many hard problems become easy in the right basis. A motion that looks messy in the standard grid may be a simple stretch in a well-chosen basis. Choosing a good basis is a quiet theme running through a lot of machine learning.

5. Everything the video skipped

Bases are not unique. The plane has infinitely many bases. The number of vectors in a basis is always the same (the dimension), but the vectors themselves are a choice.

Bases in higher dimensions. In 3D you need three independent vectors, in n dimensions you need n. The recipe is identical: put the basis vectors as columns of M; M converts new coordinates to standard coordinates, M^-1 goes back. Two vectors can never form a basis of 3D space, and four vectors in 3D are always dependent.

Basis versus span versus rank. A set of vectors that spans a space but contains redundancy is not a basis. Removing redundant vectors until the rest are independent gives a basis. This links to 1.20: the rank of a matrix counts its independent columns, which is the dimension of the space those columns span.

Why the change-of-basis matrix is always invertible. Its columns are independent (that is what makes them a basis), so its determinant is non-zero (1.19). A dependent set would squash the plane to a line, and there would be no way back.

Orthonormal bases. If the basis vectors are perpendicular and have unit length, the inverse of M is just its transpose, and finding coordinates needs only dot products. That is the topic of 1.22 (orthogonality and projections). Jen's arrows are neither perpendicular nor unit length, which is why the inverse took real work. Olive's basis in the practice problems, (2, 3) and (-3, 2), is perpendicular but not unit length.

Eigenbasis (links back to 1.15). If a matrix has enough independent eigenvectors, you can use them as a basis. In that basis the matrix is diagonal: each basis arrow is only stretched, not turned. This is the idea behind diagonalization, and SVD is a close cousin that uses two bases.

Coordinates versus the vector itself. A vector is a geometric object. A list of numbers is only its description in some basis. The same list of numbers in different bases is a different vector.

Informal links to AI (general background, not confirmed in the sources I opened). Principal component analysis chooses a new basis that follows the main directions of spread in the data. The Fourier transform describes a signal using waves as the basis. Hidden layers of a neural network can be loosely thought of as re-describing the input in new coordinates, though real layers also use nonlinear activations, so this is only an analogy. Computing with numpy is straightforward: numpy.linalg.solve solves a linear system and numpy.linalg.inv computes an inverse, both listed in the NumPy linear algebra reference. In practice, solving is usually preferred to forming an inverse explicitly, but that is a general numerical rule of thumb I did not confirm in the pages I opened.

Limits and trade-offs. A bad basis (two arrows nearly parallel) still works on paper, but the inverse involves dividing by a tiny determinant, and in real computation this makes numbers fragile. Orthonormal bases avoid the problem, which is one reason they are loved.

Where this sits in the roadmap. Before: 1.18 (matrices as motion), 1.19 (determinant and inverse), 1.20 (rank and independence). After: 1.22 (orthogonality and projections), and then back to eigenvectors and SVD from 1.15 with fresh eyes.

6. Worked examples

All numbers below were checked in Python.

Example A. Jen's coordinates to ours

Jen's basis: b1 = (2, 1), b2 = (-1, 1). Her coordinates: (-1, 2).

Multiply each coordinate by its arrow: -1 times (2, 1) = (-2, -1). 2 times (-1, 1) = (-2, 2).
Add: (-2 + -2, -1 + 2) = (-4, 1).
Matrix check: M = [[2, -1], [1, 1]]. Row 1: 2*(-1) + (-1)2 = -4. Row 2: 1(-1) + 1*2 = 1. Result (-4, 1). Same.

Another: Jen's (2, -3) gives 2*(2, 1) + (-3)*(-1, 1) = (4, 2) + (3, -3) = (7, -1). This matches the 3Blue1Brown practice answer.

Example B. Our coordinates to Jen's

Our vector: (3, 2).

det M = 2*1 - (-1)*1 = 3.
M^-1 = (1/3) [[1, 1], [-1, 2]] = [[1/3, 1/3], [-1/3, 2/3]].
M^-1 (3, 2): first entry (1/3)*3 + (1/3)*2 = 5/3. Second entry (-1/3)*3 + (2/3)*2 = -1 + 4/3 = 1/3. So Jen's coordinates are (5/3, 1/3).
Check by rebuilding: 5/3 * (2, 1) = (10/3, 5/3). 1/3 * (-1, 1) = (-1/3, 1/3). Sum: (10/3 - 1/3, 5/3 + 1/3) = (9/3, 6/3) = (3, 2). Correct.

Jen's coordinates are fractions, which is fine: her grid cells are big compared with ours, so we sit between her grid points. This matches the value stated on the 3Blue1Brown change-of-basis page.

Example C. A rotation seen in Jen's basis

R = [[0, -1], [1, 0]] turns every vector 90 degrees counterclockwise in our coordinates. We want M^-1 R M.

R M: R times each column of M. Column 1 (2, 1) goes to (-1, 2). Column 2 (-1, 1) goes to (-1, -1). So R M = [[-1, -1], [2, -1]].
M^-1 (R M): with M^-1 = (1/3) [[1, 1], [-1, 2]]. Column 1: (1/3)(1*(-1) + 12, -1(-1) + 22) = (1/3)(1, 5) = (1/3, 5/3). Column 2: (1/3)(1(-1) + 1*(-1), -1*(-1) + 2*(-1)) = (1/3)(-2, -1) = (-2/3, -1/3).
Result, by rows: [[1/3, -2/3], [5/3, -1/3]].

Sanity check: Jen's first arrow b1 in her own language is (1, 0). The matrix sends (1, 0) to its first column (1/3, 5/3). In our language that is 1/3*(2, 1) + 5/3*(-1, 1) = (2/3 - 5/3, 1/3 + 5/3) = (-1, 2), which is b1 = (2, 1) rotated 90 degrees counterclockwise. Correct.

The same motion has a different matrix, with fractions, because Jen's grid is tilted and stretched. The matrix looks messier, but it describes exactly the same rotation.

Example D. Olive's basis (from the practice questions)

Olive: b1 = (2, 3), b2 = (-3, 2). M_O = [[2, -3], [3, 2]], det = 2*2 - (-3)*3 = 13.

Olive's (-1, 2) to ours: -1*(2, 3) + 2*(-3, 2) = (-2, -3) + (-6, 4) = (-8, 1).
Our (-5, -1) to Olive's: M_O^-1 = (1/13) [[2, 3], [-3, 2]]. Then (1/13)(2*(-5) + 3*(-1), -3*(-5) + 2*(-1)) = (1/13)(-13, 13) = (-1, 1). Check: -1*(2, 3) + 1*(-3, 2) = (-5, -1).

7. Comparison table

	Standard basis	Jen's basis	Olive's basis
Basis vectors	(1, 0), (0, 1)	(2, 1), (-1, 1)	(2, 3), (-3, 2)
Matrix M (columns)	[[1, 0], [0, 1]]	[[2, -1], [1, 1]]	[[2, -3], [3, 2]]
Determinant of M	1	3	13
Perpendicular arrows?	Yes	No	Yes
Unit length?	Yes	No	No
Inverse	Itself (identity)	(1/3) [[1, 1], [-1, 2]]	(1/13) [[2, 3], [-3, 2]]
Our (-5, -1) in this basis	(-5, -1)	(-2, 1)	(-1, 1)

Quick contrast of three related ideas:

Idea	What changes	What stays the same
Change of basis	The numbers describing a vector	The vector itself
Linear transformation	The vector itself	The basis you describe it in
Conjugation M^-1 A M	The matrix	The motion it describes

8. Common mistakes and myths

Myth: the vector changes when the basis changes. It does not. Only its description changes.
Mistake: using M when you need M^-1. M turns new-basis coordinates into standard ones. M^-1 does the opposite.
Mistake: putting the basis vectors as rows. They are columns of M.
Mistake: reading the order of M^-1 A M left to right. Read it right to left, like function composition: M acts first.
Mistake: picking parallel arrows. Parallel vectors cannot be a basis. The determinant would be 0 and no inverse exists.
Myth: a basis must be perpendicular or unit length. It does not. Any independent, spanning set works. Perpendicular unit vectors are merely convenient.
Myth: the origin moves in a new basis. The origin is shared. Changing the origin would be a shift (an affine change), not a change of basis.
Mistake: assuming coordinates must be whole numbers. (5/3, 1/3) is a perfectly good address.

9. Real-life uses

Maps and graphics: a screen, a camera and a 3D object each use their own coordinate system, and software converts between them with matrices. This is the same mechanism as Jen's translation.
Robotics: each joint has its own frame of reference, and a position known in one frame is converted into another.
Data analysis: principal component analysis re-describes data along its main directions of spread (general background).
Signals and sound: the Fourier transform describes a sound as a mix of waves instead of a list of samples (general background).
Machine learning: representing inputs in a more useful set of coordinates is a recurring theme, and a dense layer multiplies by a weight matrix (the Dive into Deep Learning page notes that matrix-vector products describe how each layer's outputs are computed from the previous layer).
Solving equations: numpy.linalg.solve finds coordinates in a given basis by solving a linear system.

10. Self-check questions

What does the coordinate pair (3, 2) really mean? Three times the first basis vector plus two times the second. In the standard basis that is 3 i-hat + 2 j-hat.
What two conditions make a set of vectors a basis? They must be linearly independent and they must span the space. In the plane, two non-parallel vectors satisfy both.
Jen's basis is (2, 1) and (-1, 1). What are Jen's coordinates (-1, 2) in our system? (-4, 1), since -1*(2, 1) + 2*(-1, 1) = (-2, -1) + (-2, 2).
Write Jen's change-of-basis matrix and its determinant. M = [[2, -1], [1, 1]] with her arrows as columns. det M = 2*1 - (-1)*1 = 3.
What is M^-1, and what does it do? M^-1 = (1/3) [[1, 1], [-1, 2]]. It converts standard coordinates into Jen's coordinates.
What are our coordinates (3, 2) in Jen's language? (5/3, 1/3). Check: 5/3*(2, 1) + 1/3*(-1, 1) = (3, 2).
Why can the change-of-basis matrix always be inverted? Its columns are independent, so its determinant is not zero.
In what order do you read M^-1 R M, and what does each part do? Right to left: M translates Jen's vector to ours, R applies the motion in our language, M^-1 translates back to Jen's.
Would (2, 4) and (1, 2) work as a basis of the plane? Why or why not? No. The second is half the first, so they are parallel, they span only a line, and the determinant of the matrix with these columns is 22 - 14 = 0.
Olive's basis is (2, 3) and (-3, 2). What are Olive's coordinates (-1, 2) in our system, and our (-5, -1) in hers? Olive's (-1, 2) is (-8, 1) in ours. Our (-5, -1) is (-1, 1) in Olive's.

11. Key takeaways, go deeper, next

Key takeaways

A basis is independent vectors that span the space; the count of them is the dimension.
Coordinates are amounts of the basis vectors, so they depend on the basis.
M (basis vectors as columns) turns new coordinates into standard coordinates. M^-1 does the reverse.
To see a motion in another basis, compute M^-1 A M, reading right to left.
The vector and the motion are the same; only the numbers change.

Go deeper

Work the Olive and Jennifer problems by hand, then check with numpy.linalg.solve.
Re-do Example C with a different motion, such as a shear [[1, 1], [0, 1]].
Try a basis of your own and check that the determinant of M is not zero.
Revisit 1.15 (eigenvectors) and ask what happens if the eigenvectors are used as the basis.

Next roadmap ID: 1.22 Orthogonality and projections.

12. Sources and further reading

Change of basis, 3Blue1Brown (Essence of Linear Algebra), Grant Sanderson: Jennifer's grid, the change-of-basis matrix, the inverse, M^-1 A M and the practice problems.
Span, 3Blue1Brown, Grant Sanderson: definitions of span, linear independence and basis.
Independence, basis and dimension, MIT OpenCourseWare 18.06SC Linear Algebra, Gilbert Strang, 2011: basis and dimension statements.
Linear algebra (numpy.linalg) routines, NumPy documentation: solve, inv, eig and svd functions.
Linear Algebra, Dive into Deep Learning (d2l.ai): matrix-vector products and matrices as transformations in neural network layers.

Accuracy note: All arithmetic was verified in Python. Definitions and the Jennifer, Olive and rotation facts come from the pages listed above. Statements about PCA, the Fourier transform, solving versus inverting, and numerical fragility are general background that those pages did not confirm, and are labelled that way in the text. Figures are exact fractions, not approximations.

Search This Blog