1.06 Partial derivatives and the gradient: study notes

Learn AI by Story

1. The idea in one sentence

When a quantity depends on many inputs, a partial derivative gives the slope along one input while the others are held still, and collecting all of those slopes into a vector, the gradient, gives the direction of steepest climb, so that walking the opposite way takes every input downhill together.

2. The story from the video

Tara stands on a hillside and can walk east, north, or anywhere in between. The hill is the bowl f(x, y) = x squared + y squared, and at the point (3, 2) its height is 13. Freezing y and sliding along x gives a slope of 6; freezing x and sliding along y gives a slope of 4. Together they form the gradient [6, 4], which points uphill, so minus the gradient points downhill. Stepping 0.1 times the gradient again and again takes her from height 13 to 8.32, 5.32 and then 3.41. Real models do the same with billions of weights, and backpropagation supplies one partial derivative for each.

3. Key definitions

Term	Precise definition	Plain English
Function of several variables	A rule that takes two or more inputs, such as f(x, y), and returns one output.	A machine with several dials and one reading.
Surface	The graph of f(x, y): a height above each point (x, y) of the ground plane.	The hillside itself.
Partial derivative	The derivative of a multi-variable function with respect to one variable, treating all others as constants. Written with the curly symbol, as in the partial of f with respect to x.	The slope if you only move along one dial.
Notation: partial symbol	The curly "d" (a rounded d) replaces the plain d to signal other variables exist. Also written f sub x.	A hint that other inputs are being held still.
Holding a variable constant	Fixing its value so it behaves like an ordinary number during differentiation.	Freeze that dial.
Slice (cross-section)	The one-variable curve left after freezing the others.	Cut the hill along a straight line and look at the profile.
Gradient	The vector of all partial derivatives at a point, written with the nabla symbol (an upside-down triangle) before f, or "grad f".	The list of slopes, one per dial.
Nabla	The symbol that stands for the operator that builds the vector of partial derivatives.	The "take all the slopes" symbol.
Vector	An ordered list of numbers (see 1.02-1.04).	An arrow, or just a list.
Steepest ascent	The direction in which the function increases fastest.	The direct line uphill.
Steepest descent	The opposite of steepest ascent: minus the gradient.	The direct line downhill.
Magnitude (length) of the gradient	The square root of the sum of squares of its components.	How steep the hill is.
Directional derivative	The rate of change of f along a chosen unit-length direction; equal to the gradient dot that direction.	Slope if you walk in this particular compass direction.
Dot product	Multiply matching entries of two vectors and add (from 1.04).	A single number measuring how aligned two arrows are.
Unit vector	A vector of length 1.	A pure direction with no size.
Level set / contour line	The set of points where f has the same value; for two variables, a curve.	A line of constant height on a map.
Perpendicular (orthogonal)	At a right angle, meaning dot product zero.	Crossing at 90 degrees.
Gradient descent	Repeatedly updating parameters by subtracting a multiple of the gradient.	Walk downhill step after step.
Learning rate	The small multiplier on the gradient in each update.	Stride length.
Weight / parameter	A trainable number inside a model.	One of the model's dials.
Loss function	A score of how wrong a model is, as a function of its weights.	The hill of mistakes.
Local minimum	A point lower than all its neighbours.	A dip that is only locally the lowest.
Saddle point	A point where the gradient is zero but it is a minimum in some directions and a maximum in others.	A mountain pass or a horse saddle.
Stochastic / mini-batch gradient	A gradient estimated from a small random sample of the data.	A quick, noisy hint of the slope.
Jacobian	The matrix of all first partial derivatives of a function with several outputs.	A table of slopes for many outputs.
Hessian	The matrix of all second partial derivatives.	A table describing curvature in every direction.
Backpropagation	An algorithm that computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network.	An efficient way to get every slope at once.

4. Step-by-step explanation

Step 1: Move from one dial to many

In 1.05 a function had one input and slope was one number. Real situations have many inputs. A house price depends on size, age and distance to a station. A model's loss depends on every one of its weights. So f takes several numbers and returns one. With two inputs we can still picture it: x and y are the east and north positions on the ground, and f(x, y) is the height above that spot. The graph is a surface.

Step 2: Our landscape

Take f(x, y) = x squared + y squared. At (0, 0) the height is 0. At (1, 0) it is 1, at (0, 2) it is 4, and at (3, 2) it is 9 + 4 = 13. Every point has a height of at least zero, and the height grows as you move away from the centre in any direction. This is a round bowl with its bottom at the origin.

Step 3: Why "the slope" no longer makes sense

Stand at (3, 2). Walking east, the ground rises one way. Walking north, it rises a different amount. Walking north-east, something in between. There is no single slope, only a slope per direction. We build up from the simplest directions: the two axes.

Step 4: Freeze one variable and slice

To get the slope in the x direction, freeze y at its current value 2. The function becomes f(x) = x squared + 4, an ordinary one-variable curve. Its slope at x = 3 comes from 1.05: the derivative of x squared is 2x, and the 4 is just a constant that vanishes, giving 6. So the partial derivative with respect to x at (3, 2) is 6.

Why does freezing work? A derivative asks "what happens to the output if the input changes a tiny bit". A partial derivative asks the same question while promising that no other input changes at the same time. Freezing y makes that promise true, and it turns a many-variable problem back into a one-variable problem we already know how to solve.

Step 5: Do the same for y

Freeze x at 3. The function becomes f(y) = 9 + y squared. Its slope at y = 2 is 2y = 4. So the partial derivative with respect to y at (3, 2) is 4.

As formulas: the partial of f with respect to x is 2x, and the partial of f with respect to y is 2y. At (3, 2) these are 6 and 4. Notice each formula only involves its own variable here because x squared and y squared do not interact. When the terms are mixed (like x times y) the partial derivatives will involve both variables, as the examples below show.

Step 6: Collect them into the gradient

Put the two partial derivatives in a list:

gradient of f at (3, 2) = [6, 4]

That vector is the gradient. In general, for a function with n inputs, it has n entries. For f = x squared + y squared the gradient formula is [2x, 2y], which is just 2 times the position vector. That tells us something nice: at every point the gradient points directly away from the centre of the bowl, which is obviously the way uphill.

Step 7: Why the gradient points uphill

Each entry says how much f rises per unit step along that axis. A big positive entry says "moving this way raises f a lot". The vector [6, 4] therefore leans most towards x (6 is bigger than 4) and points up and away from the bowl's bottom. The key claim, justified in section 5, is that of all directions, this one gives the fastest rise. Therefore the opposite, minus the gradient, which is [-6, -4], gives the fastest fall.

Step 8: Walk downhill

Choose a step size of 0.1 (the learning rate). The update is:

new position = old position - 0.1 times gradient

Starting at (3, 2) with f = 13:

Gradient [6, 4]. Subtract [0.6, 0.4]. New point (2.4, 1.6). Height 5.76 + 2.56 = 8.32.
Gradient [4.8, 3.2]. Subtract [0.48, 0.32]. New point (1.92, 1.28). Height 3.6864 + 1.6384 = 5.3248, about 5.32.
Gradient [3.84, 2.56]. Subtract [0.384, 0.256]. New point (1.536, 1.024). Height 2.359296 + 1.048576 = 3.407872, about 3.41.

The heights 13, 8.32, 5.32, 3.41 fall every time, and each step is shorter than the one before because the slope flattens as we near the bottom. Every coordinate is multiplied by 0.8 at each step, since x minus 0.1 times 2x equals 0.8x.

Step 9: Scale up to real models

A modern language model has billions of weights, so its loss is a function of billions of inputs. We cannot picture that landscape, but the algebra is identical. The gradient has one entry per weight. Backpropagation (arriving in 1.07 and beyond) computes all those entries efficiently, and each training step nudges every weight a little against its own entry. Training is Tara's walk repeated millions of times in a space with billions of directions.

5. Everything the video skipped

Notation

The partial of f with respect to x is written with the curly d: the symbol, f, a line, the symbol, x. Many texts also write f sub x, or D sub x f.
The gradient is written with nabla in front of f, or as "grad f".
You may see the gradient as a column vector, a row vector, or a plain list. For our purposes they are the same list.
The point is usually indicated as nabla f evaluated at (3, 2).

Rules carry over

All rules from 1.05 (constants, power, sum, multiple) hold for partial derivatives. The only extra instruction is that other variables count as constants. Example: for f(x, y) = x squared times y, treat y as a number when differentiating with respect to x.

Gradient magnitude is the steepness

The length of the gradient is the steepest slope available. At (3, 2): the square root of (36 + 16) = square root of 52, about 7.21. The slope along the x axis is 6, along the y axis is 4, but in the best direction it is 7.21, which is greater than either.

Directional derivative

To find the slope in any direction, take the dot product (from 1.04) of the gradient with a unit-length direction vector u:

slope along u = gradient dot u

At (3, 2), gradient [6, 4]:

Direction u	Calculation	Slope
East [1, 0]	6 times 1 + 4 times 0	6
North [0, 1]	6 times 0 + 4 times 1	4
[0.8, 0.6] (length 1)	6 times 0.8 + 4 times 0.6 = 4.8 + 2.4	7.2
[0.6, 0.8] (length 1)	3.6 + 3.2	6.8
Gradient direction [6, 4] divided by 7.21, about [0.832, 0.555]	6 times 0.832 + 4 times 0.555	about 7.21
Perpendicular [-4, 6] divided by 7.21	(-24 + 24) divided by 7.21	0
Opposite [-0.832, -0.555]	negative of the best	about -7.21

The east and north slopes are just special cases of this formula, which is why partial derivatives are directional derivatives along the axes.

Why the gradient is the steepest direction

The dot product of two vectors equals the product of their lengths times the cosine of the angle between them. With u of length 1, the slope in direction u equals the gradient's length times the cosine of the angle between u and the gradient. Cosine is largest (equal to 1) when the angle is 0, meaning u points the same way as the gradient. So the best slope is the gradient's own length, and it occurs exactly along the gradient. At 90 degrees the cosine is 0, giving no change, and at 180 degrees it is -1, giving steepest descent. This is the reason the dot product from 1.04 matters so much.

Level sets and contour lines

Imagine a topographic map. Each contour line joins points of equal height. For our bowl, f = 13 is the circle x squared + y squared = 13, centred at the origin with radius about 3.61, and (3, 2) is on it. Walking along a contour gives a slope of zero (no height change), which, by the directional-derivative formula, means the direction has zero dot product with the gradient. So the gradient is perpendicular to the contour line. At (3, 2) the gradient [6, 4] is parallel to the radius [3, 2], and a radius meets a circle at a right angle, which matches the claim. Practical reading: the gradient is the arrow that cuts across contour lines the shortest way to higher ground.

Gradient descent, written as a rule

Using w for the list of weights and L for the loss:

w new = w old - learning rate times gradient of L at w old

It is an update on every weight at once. The sign matters: the gradient points uphill, so we subtract.

What a too-large learning rate does

For our bowl, gradient = 2 times the position. With learning rate 1.1, the update is new = old - 1.1 times 2 times old = old times (1 - 2.2) = -1.2 times old. Each step flips the sign and multiplies the distance by 1.2, so we jump over the bottom and land further away every time:

Start (3, 2), height 13.
Next (-3.6, -2.4), height 12.96 + 5.76 = 18.72.
Next (4.32, 2.88), height 18.6624 + 8.2944 = 26.9568.

The height grows: divergence. In general, for this bowl the update multiplies position by (1 - 2 times rate). The rate must be between 0 and 1 for the distance to shrink. Some other values:

Learning rate	Factor on position	Behaviour
0.1	0.8	Smooth, steady convergence
0.5	0	Lands exactly at the bottom in one step (special to this bowl)
1.0	-1	Bounces between opposite points forever, height stays 13
1.1	-1.2	Diverges

A tiny rate such as 0.001 has factor 0.998 and would converge very slowly. Real loss surfaces are not perfect bowls, so practitioners tune the rate, often shrinking it during training.

Local minima and saddle points

Where the gradient is the zero vector, the surface is flat in every direction, and gradient descent stalls. Such a critical point could be a minimum, a maximum or a saddle point. Consider f(x, y) = x squared - y squared: its gradient [2x, -2y] is zero at the origin, yet the surface curves up along x and down along y, like a saddle. In very high dimensions saddle points are far more common than true local minima, and practical optimisers add tricks (such as momentum) to slide past them. The second-derivative information for many variables lives in the Hessian matrix, which is the many-variable cousin of f'' from 1.05.

Stochastic and mini-batch gradients (preview)

The true loss averages mistakes over the entire dataset, which may be millions of examples. Computing the exact gradient every step is expensive. Instead, training often uses a small random batch (for example 32 or 256 examples) to estimate the gradient. The estimate is noisy, but it is cheap and on average points the right way. This is stochastic gradient descent (SGD) with mini-batches. The noise even helps escape shallow traps.

Jacobian and Hessian (a mention)

Jacobian: if a function takes n inputs and returns m outputs (a layer of a network), the matrix of all m times n partial derivatives is the Jacobian. It is the multi-output generalisation of the gradient.
Hessian: the n times n matrix of second partial derivatives, describing curvature. It is powerful but too large to store for billions of weights, which is why most training uses only the gradient.

Loss as a function of weights

A "tiny linear model" predicts y-hat = w times x + b with two weights, w and b. Its mean squared error over data is a function L(w, b), so it is a surface above the (w, b) plane. Training is finding the lowest point of that surface. The worked examples below carry this out with numbers.

Links backward and forward

Backward: 1.05 gave the one-variable derivative and the downhill rule; 1.04 gave the dot product used for directional slopes; vectors (earlier in the module) are the gradient's format.
Forward: 1.07 The chain rule: why backprop works. The loss is built by nesting functions (weights, then layers, then loss), and the chain rule tells us how to multiply local slopes along the nest to get each weight's partial derivative.

A little history

Partial derivatives were developed in the 1700s by Euler, d'Alembert and others for studying vibrating strings and fluids. The gradient idea in optimisation goes back to Cauchy's method of steepest descent (1847). Backpropagation as a practical way to get gradients through networks was popularised by Rumelhart, Hinton and Williams in 1986.

Limitations

The gradient is local: it knows the slope at your feet, not where the global bottom is.
It needs differentiable functions; sharp corners need conventions.
A good direction is not a good distance: the step size must be chosen well.
The gradient depends on how the inputs are scaled, so very differently scaled weights can make progress zig-zag.

6. Worked examples

Example A: a mixed function f(x, y) = x squared times y at (3, 2)

Height: 9 times 2 = 18.
Partial with respect to x: treat y as a constant, so 2x times y = 2 times 3 times 2 = 12.
Partial with respect to y: treat x squared as a constant multiplier, so the derivative is x squared = 9.
Gradient: [12, 9]. Magnitude: square root of (144 + 81) = square root of 225 = 15 exactly.
Numerical check on x: f(3.01, 2) = 3.01 squared times 2 = 9.0601 times 2 = 18.1202. Rise 0.1202 over run 0.01 = 12.02. Close to 12.
Numerical check on y: f(3, 2.01) = 9 times 2.01 = 18.09. Rise 0.09 over run 0.01 = 9. Exactly 9, since f is linear in y.
Steepest-descent step with rate 0.01: new point (3 - 0.12, 2 - 0.09) = (2.88, 1.91). New height 8.2944 times 1.91 = 15.842 (to 3 decimals: 8.2944 times 1.91 = 15.8423). This is lower than 18, as expected.

Example B: tiny linear model with mean squared error (illustrative data)

The model: y-hat = w times x + b. Three data points (x, y): (1, 3), (2, 5), (3, 7). They happen to lie on y = 2x + 1, so the best answer is w = 2, b = 1. The loss is the mean squared error:

L(w, b) = (1/3) times the sum of (w x + b - y) squared.

Start at w = 0, b = 0.

Predictions are all 0. Errors (prediction minus truth): -3, -5, -7.
Loss: (9 + 25 + 49) / 3 = 83 / 3, about 27.67.
Partial with respect to w: (2/3) times the sum of (error times x). Compute: (-3 times 1) + (-5 times 2) + (-7 times 3) = -3 - 10 - 21 = -34. So the partial is (2/3)(-34) = -22.67 (rounded).
Partial with respect to b: (2/3) times the sum of errors = (2/3)(-15) = -10.
Gradient: [-22.67, -10]. Both negative: raising w or b lowers the loss.
Update with rate 0.05: w = 0 + 0.05 times 22.67 = 1.133; b = 0 + 0.05 times 10 = 0.5.
New predictions: x = 1 gives 1.133 + 0.5 = 1.633; x = 2 gives 2.767; x = 3 gives 3.9. Errors: -1.367, -2.233, -3.1. Squares: 1.868, 4.988, 9.61. Sum 16.466, divided by 3 gives about 5.49.
The loss fell from about 27.67 to about 5.49 in one step, and w and b moved toward 2 and 1.

Why the formula has that shape: the derivative of (error) squared is 2 times error times the derivative of the error. The error's derivative with respect to w is x, and with respect to b is 1. That multiplication of local slopes is the chain rule, coming in 1.07.

Example C: directional derivative and a contour check

At (3, 2) on the bowl, gradient [6, 4].

Compute the slope towards [0.8, 0.6]: 6 times 0.8 + 4 times 0.6 = 4.8 + 2.4 = 7.2. Check it is a unit vector: 0.64 + 0.36 = 1.
Compare to the maximum slope, 7.21: 7.2 is very close because [0.8, 0.6] almost matches the gradient's direction [0.832, 0.555].
Slope towards [-0.6, 0.8]: -3.6 + 3.2 = -0.4. Almost flat, because this direction is close to the contour's tangent direction.
Slope towards the exact tangent [-4, 6] divided by 7.21: zero, confirming the gradient is perpendicular to the contour.

7. Comparison table or diagram

Concept	One variable (1.05)	Several variables (1.06)
Function	f(x)	f(x, y, ...)
Slope	Derivative f'(x), a number	Gradient, a vector of partials
Flat spot	f' = 0	Gradient = zero vector
Direction of change	Left or right	Any direction in the space
Downhill rule	x - rate times f'(x)	w - rate times gradient
Curvature	f''	Hessian matrix
Slope along a direction	Not needed	Gradient dot unit direction

Text picture of the bowl seen from above (contours), with the gradient at (3, 2):

 north (y)
   ^        ____  f = 13 circle
   |      /      \
 2 +-----*   <- arrow [6,4] points away from centre (uphill)
   |    / \
   |   |   |  inner circles = lower heights
   +---+---+------> east (x)
       0   3

Descent path (illustrative of the example): (3, 2), then (2.4, 1.6), then (1.92, 1.28), then (1.536, 1.024), curving straight in towards (0, 0), each step 0.8 times the previous position.

8. Common mistakes and myths

Mistake: forgetting to freeze the other variable. When differentiating x squared times y with respect to x, y must be treated as a number.
Mistake: thinking the gradient is a single number. It is a vector with one entry per input.
Mistake: walking along the gradient to reduce a loss. That climbs. Subtract it.
Myth: the gradient points at the minimum. It points in the steepest local direction, which only happens to aim at the bottom in a perfect round bowl. On stretched valleys it points mostly across the valley.
Mistake: taking east and north slopes (6 and 4) as the steepest. The best slope is 7.21, larger than both.
Mistake: using a non-unit vector in the directional derivative. The direction must have length 1, or the answer is scaled.
Myth: a zero gradient means we found the minimum. It could be a saddle point or a maximum.
Mistake: an enormous learning rate "to be fast". With rate 1.1 on our bowl the height grows each step.
Myth: backpropagation is a different kind of learning. It is only a fast method to compute the gradient, which gradient descent then uses.

9. Real-life uses

Training neural networks: each step computes the partial derivative of the loss for every weight and moves it. Image classifiers, translators and chatbots are all trained this way.
Fitting a line to data (linear regression): the loss in w and b above is exactly what is minimised, for example predicting house price from size.
Weather and engineering maps: temperature as a function of east, north and altitude; the gradient points toward the fastest warming, and heat flows in the opposite direction.
Economics: profit depends on price and advertising spend; the partial derivatives show the benefit of nudging each separately.
Navigation and robotics: path planning uses a cost landscape, with robots rolling downhill on it.
Medical imaging and recommendation systems rely on optimisation done through gradients.
Hiking itself: the gradient is the direction in which water would run uphill if it could, and the opposite is where rain flows.

10. Self-check questions

In your own words, what is a partial derivative?
For f(x, y) = x squared + y squared, what is the height at (3, 2)?
What are the two partial derivatives of that function at (3, 2)?
What is the gradient of f(x, y) = 3x + 5y at any point?
Why do we freeze the other variables?
What is the length of the gradient [6, 4], to two decimals, and what does it tell you?
A direction is [0.6, 0.8]. What is the slope of our bowl along it at (3, 2)?
Why does the gradient point in the direction of steepest ascent?
Starting at (3, 2) with step 0.1, what is the next point and its height?
For f = x squared + y squared, what happens with learning rate 1.1? And what does the gradient look like relative to a contour line?

Answers:

The slope of a multi-variable function along one input while all other inputs are held constant.
9 + 4 = 13.
2x = 6 with respect to x, and 2y = 4 with respect to y.
[3, 5], a constant vector, because the function is a flat tilted plane.
To make the question about change in one input only; it reduces the problem to a one-variable derivative we already know.
Square root of 52, about 7.21. It is the steepness of the hill in the best direction.
6 times 0.6 + 4 times 0.8 = 3.6 + 3.2 = 6.8.
The slope along a unit direction is the gradient's length times cosine of the angle to the gradient; cosine is largest at angle zero, so the best direction is the gradient itself.
(3 - 0.6, 2 - 0.4) = (2.4, 1.6), height 5.76 + 2.56 = 8.32.
Position is multiplied by -1.2 each step, so it overshoots and the height keeps rising (13, 18.72, 26.96, ...). The gradient is perpendicular to the contour line through the point.

11. Key takeaways, go-deeper list, next roadmap ID

Key takeaways:

A partial derivative is the slope along one input with the others frozen: at (3, 2) on x squared + y squared, 6 for x and 4 for y.
The gradient is the vector of all partials, [6, 4], and points toward steepest ascent; its length 7.21 is the steepness.
Slope in any direction is the gradient dot the unit direction (from 1.04).
The gradient is perpendicular to contour lines.
Gradient descent subtracts a learning rate times the gradient: heights 13, 8.32, 5.32, 3.41.
Too large a learning rate overshoots and diverges (factor -1.2 at rate 1.1).
Zero gradient might be a minimum, maximum or saddle.
With billions of weights, backpropagation delivers billions of partials; mini-batches make each step cheap.

Go deeper:

Compute the gradient of x squared + 3y squared at (1, 1) and check it against a finite-difference test.
Draw contour lines for x squared + y squared and for x squared + 4y squared, and see how the gradient changes.
Run gradient descent by hand on L(w, b) from Example B for two more steps.
Learn about the Hessian and why saddle points matter in deep learning.
Read about momentum and Adam, popular improvements to plain gradient descent.

Next roadmap ID: 1.07 The chain rule: why backprop works.

Partial derivatives and the gradient: study notes