1.25 Jacobian and Hessian: study notes

Learn AI by Story

1. The idea in one sentence

The Jacobian is the table of "how much does each output change when I nudge each input", and the Hessian is the table of "how is the slope itself changing", which together describe the local tilt and the local bend of any function with many inputs.

2. The story from the video

Omar has a sound mixer with two knobs and two speakers. Turning the first knob changes the loudness of both speakers by some amounts, and turning the second knob changes them by other amounts, so there are 2 x 2 = 4 "how much" numbers. Omar writes them in a small grid: that grid is the Jacobian, and it predicts what happens to both speakers for a small twist of both knobs. Then he asks a second question: if I keep turning one knob, does the effect get stronger or weaker? That "change of the change" lives in another grid, the Hessian, and it tells him whether he is at the bottom of a bowl (best sound), the top of a hill, or on a saddle where one knob helps and the other hurts.

3. Key definitions (glossary)

Partial derivative: the rate of change of a function with respect to one input while all other inputs are held still. Written ∂f/∂x. For f = xy, ∂f/∂x = y.
Gradient: for a function with one output and many inputs, the vector of all its partial derivatives, written ∇f = (∂f/∂x, ∂f/∂y, ...). It points in the direction of steepest increase.
Jacobian: for a function with several outputs and several inputs, the matrix J whose entry in row i, column j is ∂fᵢ/∂xⱼ. With m outputs and n inputs it is m x n. For one output it is just the gradient written as a row.
Determinant of the Jacobian: for a square Jacobian, det(J) tells you how much the function locally stretches or shrinks areas (or volumes), and a negative sign means it flips orientation. This is the same determinant as in 1.19, applied to the local linear map.
Hessian: for a function with one output, the n x n matrix H of all second partial derivatives, Hᵢⱼ = ∂²f/∂xᵢ∂xⱼ. It is the Jacobian of the gradient.
Symmetric: a matrix equal to its own transpose (Hᵢⱼ = Hⱼᵢ). The Hessian is symmetric whenever the second derivatives are continuous, because the order of differentiation does not matter.
Eigenvalue (of the Hessian): a number that says how much the function curves along a special direction (an eigenvector). Positive means curving up, negative means curving down.
Positive definite: a symmetric matrix whose eigenvalues are all positive. A positive definite Hessian at a flat point means a bowl, so a local minimum.
Saddle point: a flat point (gradient zero) that is a bowl in one direction and a hill in another, such as f = x² − y² at the origin. The Hessian has one positive and one negative eigenvalue.
Curvature: how fast the slope changes. A steep bowl has high curvature, a flat plain has low curvature.
Condition number: the ratio of the largest to the smallest eigenvalue of the Hessian (for a positive definite one). A big ratio means a long thin valley, where gradient descent zigzags and is slow.
Newton's method: an optimisation step that uses both the gradient and the Hessian: x_new = x − H⁻¹∇f. It jumps straight to the bottom of the local quadratic bowl.
Hessian-vector product: the product Hv of the Hessian with a vector v. It can be computed without ever writing down H, at about the cost of a couple of gradient evaluations, which makes second-order ideas affordable for big models.
Chain rule: how derivatives combine through composition. For f(g(x)) the Jacobian is the product of the Jacobians, J_f(g(x)) times J_g(x). Backpropagation is the chain rule applied layer by layer.

4. Detailed step-by-step explanation

Step 1: one input, one output (the slope)

For y = f(x), the derivative f'(x) is the slope of the tangent line. For a small step h, f(x + h) is roughly f(x) + f'(x) h. That is the whole idea of local linearisation: close to a point, a curve looks like a straight line.

Step 2: many inputs, one output (the gradient)

Now let f depend on x and y. Nudge x alone and measure the response: that is ∂f/∂x. Nudge y alone: ∂f/∂y. Stack them and you get the gradient. For a small step (h, k), f changes by about (∂f/∂x) h + (∂f/∂y) k, which is the dot product ∇f · (h, k). The gradient tells you the best direction to walk uphill; negative gradient is the direction gradient descent uses.

Step 3: many inputs, many outputs (the Jacobian)

Omar's mixer has two knobs (inputs x, y) and two speakers (outputs f₁, f₂). Each speaker has its own gradient. Stack the two gradients as rows and you get the Jacobian:

J = [[∂f₁/∂x, ∂f₁/∂y], [∂f₂/∂x, ∂f₂/∂y]]

Row i answers "how does output i react to each knob", column j answers "how do all outputs react to knob j". For a small twist of the knobs by a vector Δ, the outputs change by about J Δ. That is a matrix times a vector, so the Jacobian is the best linear approximation of the function near the point, which links back to 1.18 linear transformations.

Step 4: what the Jacobian determinant says

A small square of knob settings is mapped to a small parallelogram of speaker levels. The area of that parallelogram, relative to the square, is |det J|. If det J is 0, the map squashes the square to a line and information is lost. Normalising flows and the change-of-variables formula in probability use exactly this factor.

Step 5: the second question (curvature)

The gradient only tells you the tilt. Imagine standing on a hillside: knowing the slope does not say whether the ground ahead curves up (a valley) or down (a ridge). The second derivatives answer that. For one input, f'' is the bend. For many inputs there are many ways to bend: ∂²f/∂x², ∂²f/∂y², and the cross terms ∂²f/∂x∂y. Collect them in a matrix and you get the Hessian.

Step 6: reading the Hessian with eigenvalues

Because H is symmetric, it has real eigenvalues and perpendicular eigenvector directions. Walk along an eigenvector and the function bends by the eigenvalue. All positive: a bowl. All negative: a dome. Mixed signs: a saddle. Some zero: a flat direction where this test is inconclusive. This is why the Hessian is the classic tool for classifying a point where the gradient is zero.

Step 7: why it matters for gradient descent

The second-order Taylor idea (previewed here, covered in 1.26) says: f(x + s) ≈ f(x) + gᵀs + ½ sᵀHs, where g is the gradient. If you step downhill with s = −εg, the loss changes by about −ε gᵀg + ½ ε² gᵀHg. The first term is progress; the second term is the cost of curvature. Make ε too big and the second term wins and you go uphill. So the biggest safe learning rate is tied to the largest eigenvalue of the Hessian.

Step 8: Newton's method

If you trust the local bowl, you can jump to its bottom in one move: s = −H⁻¹g. For a perfectly quadratic function it lands exactly on the minimum in one step. The price is computing and inverting H, which has n² entries and costs about n³ operations to solve. That is fine for 2 or 200 parameters and impossible for a million.

5. Everything the video skipped

Gradient is a special Jacobian. One output means a 1 x n Jacobian, i.e. the gradient written as a row. Some books use a column for the gradient; the meaning is identical, only the layout differs.

Shapes. Function from n inputs to m outputs: Jacobian m x n. Scalar function of n inputs: gradient n, Hessian n x n. If a function has m outputs, its second derivatives form an m x n x n tensor (one Hessian per output). This is the tensor link to 1.24.

Hessian = Jacobian of the gradient. Take the gradient (a vector function of x) and compute its Jacobian: you get the Hessian. That is why it is symmetric for smooth functions.

Second derivative test. At a point where ∇f = 0: H positive definite means local minimum; negative definite means local maximum; mixed signs means saddle; zero eigenvalues means the test is inconclusive.

Saddles dominate in high dimensions. With millions of parameters, a random flat point has to be a bowl in every direction to be a minimum, which is unlikely. So saddle points (and wide flat regions) are far more common than bad local minima in neural network loss surfaces. Goodfellow's book discusses this at length.

Condition number. If the largest eigenvalue is 100 and the smallest is 1, the condition number is 100. The valley is steep in one direction and shallow in another. Gradient descent must use a small step for the steep direction and then crawls along the shallow direction. Newton's method (and approximations such as preconditioning, Adam-style scaling, or quasi-Newton methods like L-BFGS) fixes this by rescaling directions.

Cost of the Hessian. For one million weights the Hessian has 10⁶ x 10⁶ = 10¹² entries. At 4 bytes each that is about 4 terabytes, so nobody stores it. Instead, we use Hessian-vector products (via automatic differentiation, differentiating the gradient along a direction v), power iteration to estimate the top eigenvalue, or diagonal and low-rank approximations.

Jacobian for backprop. In a network, each layer is a function, and the chain rule multiplies the Jacobians of layers. Backpropagation never builds the full Jacobians; it multiplies a vector by them (a vector-Jacobian product), which is far cheaper. Forward-mode differentiation does the opposite (Jacobian-vector products).

Exactness. The Jacobian prediction J Δ is exact only when the function is linear. For curved functions it is a good guess for small steps and drifts as the step grows. The Hessian is what describes the first correction.

Limits. Jacobians and Hessians need the function to be differentiable. Functions like ReLU have kinks, so the second derivative is zero almost everywhere and undefined at the kink. Noisy mini-batch losses make the Hessian noisy too. Newton's method also misbehaves far from a minimum or at a saddle, where H⁻¹ may point uphill.

Roadmap links. 1.18 linear transformations (the Jacobian is a local linear map); 1.19 determinant (area scaling); 1.21 basis (eigenvectors of H form a good basis for curvature); 1.24 tensors (higher-order derivative tensors); next 1.26 Taylor approximation, which formalises "tangent plus bend". Later: backpropagation (chain of Jacobians) and optimisers (Adam, second-order methods).

6. Worked examples

Example A: the video's Jacobian

Let f(x, y) = (xy, x + y²) at the point (2, 3).

f₁ = xy: ∂f₁/∂x = y = 3 and ∂f₁/∂y = x = 2.
f₂ = x + y²: ∂f₂/∂x = 1 and ∂f₂/∂y = 2y = 6.
Jacobian J = [[3, 2], [1, 6]].
Output at (2, 3): (6, 11).
Nudge x by 0.1: f(2.1, 3) = (6.3, 11.1). The prediction is f(2,3) + J (0.1, 0) = (6, 11) + (0.3, 0.1) = (6.3, 11.1). It matches exactly here because both outputs are linear in x when y is fixed.
det J = 3 x 6 − 2 x 1 = 16, so small areas are stretched by about 16 times.

Example B: the video's Hessian

Let f(x, y) = x² + 3xy + 5y².

∂f/∂x = 2x + 3y and ∂f/∂y = 3x + 10y.
Second derivatives: ∂²f/∂x² = 2, ∂²f/∂y² = 10, ∂²f/∂x∂y = 3 (and ∂²f/∂y∂x = 3, symmetric).
H = [[2, 3], [3, 10]].
Eigenvalues: solve λ² − 12λ + (20 − 9) = 0, i.e. λ² − 12λ + 11 = 0, giving λ = 1 and λ = 11. Both positive, so H is positive definite and the flat point at (0, 0) is a minimum (a bowl).
det H = 2 x 10 − 3 x 3 = 11, equal to 1 x 11. Condition number 11/1 = 11.
Contrast with the saddle f = x² − y²: H = [[2, 0], [0, −2]], eigenvalues 2 and −2, mixed signs, so a saddle.

Example C (new): a Jacobian with a nonlinear change

Let f(x, y) = (x²y, xy²) at the point (2, 1).

∂f₁/∂x = 2xy = 4, ∂f₁/∂y = x² = 4.
∂f₂/∂x = y² = 1, ∂f₂/∂y = 2xy = 4.
J = [[4, 4], [1, 4]], det J = 16 − 4 = 12.
f(2, 1) = (4, 2). Nudge x to 2.1: f(2.1, 1) = (4.41, 2.1). Real change (0.41, 0.1).
Jacobian prediction: J (0.1, 0) = (0.4, 0.1). First output is off by 0.01 because x²y curves in x; second output is exact.
Nudge both by 0.1: real change f(2.1, 1.1) − f(2, 1) = (0.851, 0.541); prediction J (0.1, 0.1) = (0.8, 0.5). The error is a bit bigger, as expected when the step is bigger.

These numbers were checked by computing them in NumPy.

Example D (new): Newton's method on the video's bowl

Take f = x² + 3xy + 5y² and start at (1, 1).

Gradient g = (2 + 3, 3 + 10) = (5, 13).
Solve H s = g with H = [[2, 3], [3, 10]]: the solution is s = (1, 1), because 2 + 3 = 5 and 3 + 10 = 13.
Newton step: (1, 1) − (1, 1) = (0, 0), the exact minimum, in one step.
Gradient descent with a safe step ε below 2/11 (about 0.18, set by the largest eigenvalue 11) would need many steps, since the other direction has curvature only 1.

Example E (new): a diagonal Hessian

For f = x² + 50y², H = [[2, 0], [0, 100]], eigenvalues 2 and 100, condition number 50. The valley is 50 times steeper in y than in x, so gradient descent zigzags. The eigenvalues were checked with NumPy.

7. Comparison tables

Derivative objects

Object	Function type	Shape	Tells you
Derivative	1 input, 1 output	scalar	slope
Gradient	n inputs, 1 output	n	steepest-uphill direction
Jacobian	n inputs, m outputs	m x n	best linear map (tilt)
Hessian	n inputs, 1 output	n x n	curvature (bend)
Hessian of vector function	n inputs, m outputs	m x n x n	one bend table per output

Reading the Hessian at a flat point

Eigenvalues	Name	Picture	Verdict
all positive	positive definite	bowl	local minimum
all negative	negative definite	dome	local maximum
mixed signs	indefinite	saddle	neither
some zero	semi-definite	flat direction	test inconclusive

Gradient descent vs Newton

Property	Gradient descent	Newton's method
Uses	gradient only	gradient and Hessian
Cost per step	cheap (n)	expensive (about n³)
Memory	n	n²
Steps needed	many on thin valleys	few near a minimum
Risk	slow, learning-rate tuning	wrong direction away from a minimum, singular H

8. Common mistakes and myths

Mistake: mixing up rows and columns. Jacobian entry (i, j) is output i with respect to input j, so shape is outputs x inputs.
Mistake: calling the Jacobian the gradient. They agree only for a single output.
Myth: the Jacobian is the exact answer. It is the best straight-line guess; it is exact only for linear functions.
Mistake: assuming zero gradient means minimum. It could be a maximum or a saddle; check the Hessian.
Myth: bad local minima are the main trouble in deep learning. In high dimensions saddles and flat regions are far more common.
Mistake: thinking the Hessian is cheap. n parameters means n² entries.
Mistake: forgetting symmetry assumptions. The Hessian is symmetric when second derivatives are continuous; for non-smooth functions be careful.
Mistake: using Newton far from the minimum. The quadratic picture may be wrong there, and the step can go uphill.
Mistake: confusing "determinant of Jacobian" with "determinant of Hessian". The first is area scaling of a map; the second multiplies the curvatures.

9. Real-life and AI uses

Backpropagation: the chain rule multiplies layer Jacobians (as vector-Jacobian products) to get the gradient of the loss.
Optimisation: learning rates, momentum and Adam all try to cope with curvature; Newton and L-BFGS use it directly.
Second-order methods in practice: K-FAC and Shampoo-style optimisers approximate curvature for large networks.
Loss landscape analysis: the top Hessian eigenvalue is used to measure how sharp or flat a minimum is, which is studied in relation to generalisation.
Normalising flows and change of variables: the log determinant of the Jacobian corrects probabilities when a variable is transformed.
Robotics and control: the Jacobian relates joint angles to hand movement.
Statistics: the Hessian of the negative log-likelihood gives the uncertainty of fitted parameters (the Fisher information is closely related).
Model sensitivity and adversarial examples: the Jacobian of the output with respect to the input shows which input pixels matter most.

10. Self-check questions

What shape is the Jacobian of a function with 3 inputs and 2 outputs? 2 x 3.
What is the Jacobian of f = (xy, x + y²) at (2, 3)? [[3, 2], [1, 6]].
What does f(2.1, 3) equal, and what does the Jacobian predict? (6.3, 11.1) both.
What is the Hessian of x² + 3xy + 5y²? [[2, 3], [3, 10]].
What are its eigenvalues and what do they say? 1 and 11, both positive, so a bowl and a minimum.
Why is the Hessian symmetric? For smooth functions the order of taking partial derivatives does not matter.
What is a saddle point, with an example? A flat point that is a minimum in some directions and a maximum in others, like x² − y² at the origin.
How many Hessian entries does a model with 10⁶ weights have? 10¹².
What is the Newton step formula? x_new = x − H⁻¹∇f.
What does a large condition number do to gradient descent? It makes progress slow and zigzagging because the valley is steep in one direction and shallow in another.

11. Key takeaways, go deeper, next

Key takeaways

Jacobian: outputs x inputs table of first derivatives; the best local linear map.
Hessian: inputs x inputs table of second derivatives; the local bend, symmetric for smooth functions.
Eigenvalues of the Hessian classify flat points: bowl, dome or saddle.
Curvature limits the learning rate and slows gradient descent when the condition number is large.
Newton's method uses H but is too costly for large models; Hessian-vector products give a cheap way to use curvature.
Backpropagation is the chain rule: a product of Jacobians.

Go deeper

Compute the Jacobian of your own two-input, two-output function and test the prediction with a tiny nudge.
Use numpy.linalg.eigvalsh on a few 2 x 2 symmetric matrices and sketch the matching surface.
Run gradient descent and Newton's method on x² + 50y² and compare steps.
Read the section on Jacobian and Hessian matrices in Goodfellow's chapter 4.

Next on the roadmap: 1.26 Taylor approximation, which combines the tangent (gradient or Jacobian) and the bend (Hessian) into one formula.

12. Sources and further reading

Accuracy note: the mixer story, f = (xy, x + y²) at (2, 3), the quadratic x² + 3xy + 5y² and the extra examples are illustrative; all numbers were checked with NumPy, and the 4 TB figure assumes 4-byte floats.

Search This Blog