1.17 Vector norms: L1, L2, L-infinity: study notes

Learn AI by Story

1. The idea in one sentence

A vector norm is a rule that turns an arrow into a single non-negative number that says "how big is it", and the three most useful rules are L1 (add up the steps), L2 (the straight-line length) and L-infinity (the single biggest step), each of which measures size differently and so makes a machine-learning model behave differently.

2. The story from the video

Kai the courier walks 3 blocks east and 4 blocks north. If Kai must follow the streets, the trip is 3 + 4 = 7 blocks. If a drone flies straight from start to finish, it covers 5 blocks, because 3, 4 and 5 form a right triangle. And if Kai only cares about the longest single leg, the answer is 4. Same trip, three honest answers: 7, 5 and 4. Each answer is a norm, and which one is "right" depends on what you are trying to measure.

3. Key definitions

Every term below has a precise meaning first and a plain meaning after it.

Vector: an ordered list of numbers, such as (3, 4), read as an arrow from the origin. Plain meaning: a trip described by how far to go along each direction.
Component (entry): one number inside a vector. Plain meaning: one leg of the trip.
Absolute value |x|: the size of a number with its sign removed, so |-5| = 5. Plain meaning: distance from zero, ignoring direction.
Norm: a function that assigns every vector a non-negative number and obeys three rules (positive definiteness, absolute homogeneity, triangle inequality). Plain meaning: an agreed way to measure size.
Positive definiteness: the norm is never negative and is zero only for the zero vector. Plain meaning: only "no trip at all" has size zero.
Absolute homogeneity: scaling a vector by a number a scales its norm by |a|. Plain meaning: double the trip, double the size; reverse it, same size.
Triangle inequality: the norm of a sum is at most the sum of the norms. Plain meaning: going somewhere in two legs is never shorter than going directly.
L1 norm (also called the taxicab or Manhattan norm): the sum of the absolute values of the entries. Plain meaning: blocks walked along the grid.
L2 norm (the Euclidean norm): the square root of the sum of the squared entries. Plain meaning: the straight-line length.
L-infinity norm (the max norm): the largest absolute value among the entries. Plain meaning: the longest single leg.
p-norm (Lp): for p of at least 1, the p-th root of the sum of |entry| raised to the power p. Plain meaning: one dial, p, that slides between the three norms.
Squared L2 norm: the L2 norm multiplied by itself, which is just the sum of the squared entries (no square root). Plain meaning: L2 without the final root; cheaper and smooth.
Unit ball: the set of all vectors whose norm is at most 1 (its boundary is the "unit sphere"). Plain meaning: the shape you get if you may travel exactly one unit in the chosen measure.
Quasi-norm: a size measure that fails one norm rule. For p below 1, the unit shape has concave sides and the triangle inequality breaks. Plain meaning: norm-like but not a true norm.
"L0" count: the number of non-zero entries. NumPy's documentation notes that orders below 1 are not strictly a mathematical norm. Plain meaning: how many entries are switched on.
Frobenius norm: the matrix analogue of L2, the square root of the sum of all squared entries of a matrix. Plain meaning: L2 for a table of numbers.
Regularization: adding a penalty on the size of the model's weights to the training objective. Plain meaning: a tax on big or numerous weights.
Sparse: mostly zeros. Plain meaning: most switches are off.
Lasso: regression with an L1 penalty on the coefficients. Ridge: regression with a squared-L2 penalty. Plain meaning: two different taxes on weights.
Adversarial example: an input altered by a small, deliberately chosen perturbation so a model misclassifies it. Plain meaning: a nudge too small for a person to notice that still fools the model.

4. Step-by-step explanation

Step 1: Why we need a measure of size at all

A single number is easy to compare, but a vector holds several. To say "this error vector is small" or "these weights are too large", we must collapse a list into one number. That collapse is a norm.

Step 2: L1, add up the legs

Take the absolute value of each entry and add them. For (3, 4): |3| + |4| = 7. We use absolute values because walking west does not cancel walking east; you still spent the shoe leather. Both the course page on norms (apxml) and the Manhattan-distance description in the d2l textbook describe L1 as distance along the axes, like city streets.

Step 3: L2, the straight line

Square each entry, add, take the square root. For (3, 4): the square root of 9 + 16 is the square root of 25, which is 5. This is the Pythagorean theorem, and it is the length a ruler would show. It is the default in NumPy: np.linalg.norm(v) with no extra argument computes L2.

Step 4: L-infinity, the biggest leg

Take the largest absolute entry. For (3, 4) that is 4. This measure ignores everything except the worst coordinate. It is the right tool when you need a guarantee that no single coordinate exceeds a limit.

Step 5: One formula for all three

The p-norm is (|x1|^p + |x2|^p + ... + |xn|^p) raised to 1/p. Put p = 1 and you get L1. Put p = 2 and you get L2. As p grows without bound, the biggest entry dominates everything else and the result approaches the largest absolute entry, which is L-infinity. NumPy's documentation uses exactly this general formula, sum(abs(x)ord)(1/ord), and gives the max for ord = inf.

Step 6: The ordering rule

For any vector, L-infinity is at most L2, which is at most L1. For (3, 4): 4 <= 5 <= 7. For (-5, 12): L1 is 5 + 12 = 17, L2 is the square root of 25 + 144 = 169, which is 13, and L-infinity is 12; so 12 <= 13 <= 17. The ordering makes sense: the biggest leg alone is no more than the straight line, and the straight line is no more than the full walk. Equality happens when only one entry is non-zero.

Step 7: Dimension bounds

How different can the norms be? For a vector with n entries, L1 is at most the square root of n times L2, and L2 is at most the square root of n times L-infinity. The vector of four ones shows the extreme case: L1 = 4, L2 = 2, L-infinity = 1. Here 4 = sqrt(4) times 2 and 2 = sqrt(4) times 1, so both bounds are tight. The constants grow with n, so in very long vectors the three norms can disagree strongly.

Step 8: Unit balls and why their shapes matter

In two dimensions the set of points with norm at most 1 is a diamond for L1, a circle for L2 and a square for L-infinity. The diamond has sharp corners sitting exactly on the axes, where one coordinate is zero. This is the geometric reason L1 constraints tend to produce sparse solutions: when a smooth shape such as the loss contour first touches a diamond, it often touches a corner, where some coordinate is exactly zero. The circle has no corners and no preferred direction, so it shrinks weights smoothly without zeroing them.

Step 9: The norms in training

Squared error (mean squared error) is the squared L2 norm of the residual vector, so least squares is an L2 method.
Ridge regression adds a squared-L2 penalty on the weights. scikit-learn writes the ridge objective as the minimum over w of ||Xw - y||^2 plus alpha times ||w||^2, and notes that larger alpha means more shrinkage and coefficients more robust to collinearity.
Lasso adds an L1 penalty. Tibshirani's 1996 paper describes minimizing the residual sum of squares subject to the sum of the absolute values of the coefficients being less than a constant, and says this tends to produce some coefficients that are exactly 0.
Adversarial robustness often measures how far an input may be nudged. A cap on the largest per-pixel change is an L-infinity budget. This is commonly how such attacks are framed (the Goodfellow, Shlens and Szegedy paper itself attributes the vulnerability to the linear nature of neural networks, and does not make the budget claim in its abstract).

5. Everything the video skipped

Distance is a norm of a difference

The distance between two points a and b is the norm of the vector a - b. Choose L1 and you get taxicab distance; choose L2 and you get the usual distance. This is why "which norm" also decides what "nearby" means in clustering and nearest-neighbour methods.

The triangle inequality, with numbers

Take u = (3, 4) and w = (-1, 2); their sum is (2, 6). In L1: 8 versus 7 + 3 = 10. In L2: about 6.32 versus 5 + about 2.24, about 7.24. In L-infinity: 6 versus 4 + 2 = 6 (equality). All three obey "sum is at most the total of the parts".

Homogeneity, with numbers

Scale (3, 4) by 3 to get (9, 12): L1 is 21, which is 3 times 7. Scale by -2 to get (-6, -8): L2 is 10, which is 2 times 5. The sign never matters, only the magnitude of the scale.

p between 1 and infinity, and p below 1

Any p of at least 1 gives a true norm. For p = 0.5 the unit shape becomes a star with concave sides and the triangle inequality fails, so it is only a quasi-norm. The count of non-zero entries (often called L0) is likewise not a true norm, but it measures sparsity directly. L1 is the closest convex stand-in for that count, which is a major reason lasso is popular.

Matrix norms

The Frobenius norm treats a matrix as one long vector and takes its L2 norm. Other matrix norms exist, such as the spectral norm, which d2l defers to later; it connects to singular values (see 1.15 Eigenvectors and SVD).

Sensitivity to outliers

The d2l textbook notes that L1 is less sensitive to outliers than L2. The reason is arithmetic: squaring makes one large error count far more than one small error, while absolute value counts it only in proportion.

Limits and trade-offs

L1 is not smooth at zero (its graph has a corner), so gradient methods need care; L2 squared is smooth everywhere.
L2 penalties shrink but rarely zero out weights; L1 penalties can zero them but may behave unstably when features are strongly correlated (this second point is a general caution, not a claim from the sources above).
L-infinity looks only at the worst entry, so it can hide how many entries are large.
Choosing a norm is a modelling decision, not a fact of nature.

Roadmap links

Earlier: 1.14 Cross-entropy and KL (another way to measure "how far apart"), 1.15 Eigenvectors and SVD (matrix size via singular values), 1.16 Loss landscapes (penalties reshape the landscape). Later: 1.18 Linear transformations (matrices move vectors, and norms tell us how much), 1.22 Orthogonality and projections (L2 is the norm that comes from the dot product).

6. Worked examples

Example A: three norms of two vectors

Vector (3, 4):

L1 = |3| + |4| = 7.
L2 = sqrt(9 + 16) = sqrt(25) = 5.
L-infinity = max(3, 4) = 4.
Check ordering: 4 <= 5 <= 7. Correct.

Vector (-5, 12):

L1 = 5 + 12 = 17.
L2 = sqrt(25 + 144) = sqrt(169) = 13.
L-infinity = 12.
Check ordering: 12 <= 13 <= 17. Correct. (This is the 5-12-13 right triangle.)

Now a three-entry vector, (1, -2, 2): L1 = 1 + 2 + 2 = 5, L2 = sqrt(1 + 4 + 4) = 3, L-infinity = 2. Ordering 2 <= 3 <= 5 holds, and L1 is within sqrt(3) times L2 (5 is at most about 5.20).

Example B: which penalty prefers which weights (illustrative)

Two models have four weights each. Model P puts all its weight on one feature: (1, 0, 0, 0). Model Q spreads it: (0.5, 0.5, 0, 0).

Model	Weights	L1 penalty	Squared-L2 penalty
P	(1, 0, 0, 0)	1	1
Q	(0.5, 0.5, 0, 0)	1	0.5

Step by step: the L1 penalty for P is 1; for Q it is 0.5 + 0.5 = 1. A tie. The squared-L2 penalty for P is 1 squared = 1; for Q it is 0.25 + 0.25 = 0.5. So squared L2 prefers the spread-out model Q, because squaring punishes one big weight more than two medium ones. L1 does not care how the total is divided, so it has no reason to spread weight, and combined with the diamond's corners this is why it tends to land on solutions with exact zeros. The numbers are illustrative; real fits depend on the data.

Example C: a four-pixel perturbation budget (illustrative)

Suppose an image perturbation changes four pixels by (0, -6, 8, 0) intensity steps. L1 = 14, L2 = 10 (sqrt of 36 + 64 = 100), L-infinity = 8. A rule that says "no pixel may change by more than 8" is satisfied exactly; a rule that says "total L2 change at most 9" is violated. The same change passes one budget and fails another, which is why attack papers always state which norm they use.

7. Comparison table

Property	L1	L2	L-infinity
Formula	sum of absolute entries	sqrt of sum of squares	largest absolute entry
Value for (3, 4)	7	5	4
Value for (-5, 12)	17	13	12
Unit ball in 2D	diamond	circle	square
Everyday picture	blocks along streets	straight-line flight	longest single leg
Regularization use	lasso, tends to give exact zeros	ridge (squared), shrinks smoothly	rarely used as a penalty
Typical other use	robust errors, sparsity	squared error, distances	per-pixel attack budgets
Smooth everywhere?	no, corner at zero	yes (squared form)	no
NumPy call	np.linalg.norm(v, ord=1)	np.linalg.norm(v)	np.linalg.norm(v, np.inf)

8. Common mistakes and myths

"Norm" means length, always the straight line. No. Length is the L2 norm only. L1 and L-infinity are equally valid norms.
Forgetting absolute values in L1. Adding (-3) + 4 gives 1, not 7.
Forgetting the root in L2. The sum of squares, 25, is the squared L2 norm, not the L2 norm.
Believing L1 and L2 order can flip. For one vector the order L-infinity <= L2 <= L1 always holds.
Thinking ridge makes weights exactly zero. It shrinks them toward zero; lasso is the one known for exact zeros.
Thinking the L2 penalty and squared error are the same thing. Both use L2 but on different objects: errors versus weights.
Comparing norms across different lengths. A larger vector can have a larger L1 even if each entry is small; the sqrt(n) bounds show how the scale changes.
Treating the "L0 norm" as a real norm. It counts non-zeros and fails absolute homogeneity (scaling by 3 does not triple a count).

9. Real-life uses

Feature selection with lasso: switching off weak inputs in a regression so the final model uses fewer variables.
Ridge regression: stabilizing a fit when inputs are strongly correlated.
Mean squared error: the standard training loss for many regression problems.
Adversarial testing: checking how a classifier behaves when pixels are changed within an L-infinity budget.
Maps and logistics: grid-based cities are measured in L1; air travel in L2.
Nearest-neighbour search and clustering: the choice of norm decides who counts as a neighbour.
Quality control: an L-infinity tolerance says no single measurement may be off by more than a limit.

10. Self-check questions

What are L1, L2 and L-infinity of (3, 4)? Answer: 7, 5 and 4.
What are they for (-5, 12)? Answer: 17, 13 and 12.
State the ordering that always holds. Answer: L-infinity <= L2 <= L1.
Name the three rules every norm must obey. Answer: positive definiteness, absolute homogeneity, triangle inequality.
What shapes are the 2D unit balls for L1, L2 and L-infinity? Answer: diamond, circle, square.
For weights (1, 0, 0, 0) and (0.5, 0.5, 0, 0), compare the L1 and squared-L2 penalties. Answer: L1 is 1 and 1; squared L2 is 1 and 0.5, so squared L2 prefers the spread-out one.
Why does lasso tend to give exact zeros? Answer: the L1 diamond has corners on the axes where some coordinates are zero, and the constrained fit often lands on a corner.
What is the L1 norm of (-4, 4, -4)? Answer: 12. And its L-infinity? Answer: 4.
Which norm would you use to promise that no single pixel changes by more than a set amount? Answer: L-infinity.
Why is the count of non-zero entries not a true norm? Answer: scaling a vector by a non-zero number does not change the count, but absolute homogeneity requires it to scale by the absolute value of that number.

11. Key takeaways, go deeper, next

Key takeaways

A norm turns a vector into a size; L1 adds, L2 uses Pythagoras, L-infinity takes the maximum.
L-infinity <= L2 <= L1 for every vector.
The unit-ball shape (diamond, circle, square) explains the behaviour of the penalty.
Lasso uses L1 and tends to zero out weights; ridge uses squared L2 and shrinks smoothly.
The right norm depends on what you want to protect or measure.

Go deeper

Try np.linalg.norm with ord=1, 2 and np.inf on random vectors and check the ordering yourself.
Plot the unit ball for p = 0.5, 1, 2, 4 and watch it morph from star to square.
Read how scikit-learn defines ridge and lasso objectives side by side.
Explore matrix norms (Frobenius, spectral) after revisiting 1.15.

Next roadmap ID: 1.18 Linear transformations: matrices as motion.

12. Sources and further reading

Accuracy note: The norm definitions, ordering, dimension bounds, unit-ball shapes, ridge and lasso objectives and the lasso abstract wording were confirmed in the sources above, and all numbers were verified by computation. The Example B and C figures are illustrative, and the remarks on L1 behaviour with correlated features and on L-infinity attack budgets are general practice rather than claims taken from a specific source.

Vector norms: L1, L2, L-infinity: study notes