1.07 The chain rule: study notes

Learn AI by Story

1. The idea in one sentence

When one quantity depends on a second quantity, and the second depends on a third, the slope of the whole chain is the slope of each link multiplied together.

2. The story from the video

Nia turns the crank of a bicycle. The crank drives a chain, and the chain drives the wheel. She wants to know: if she turns the crank a tiny bit more, how much faster does the wheel spin? She cannot jump straight from crank to wheel. She has to pass through the middle link. Each link has its own "gearing" (its own slope), and the total effect is the first gearing times the second. In the video this became u = 3x + 1 and y = u². At x = 1 the first link has slope 3, the second link has slope 8, and the whole chain has slope 8 × 3 = 24.

3. Key definitions

Term	Precise definition	Plain-English meaning
Function	A rule that takes an input and gives exactly one output.	A machine: number in, number out.
Composite function	A function built by feeding the output of one function into another, written f(g(x)).	Two machines connected in a row.
Inner function	The function applied first, g in f(g(x)).	The first link of the chain.
Outer function	The function applied last, f in f(g(x)).	The final link of the chain.
Intermediate variable	The name given to the inner output, such as u = g(x).	The middle value passed along the chain.
Derivative	The instantaneous rate of change of a function at a point; the slope of its tangent line.	How steeply the output moves per tiny nudge of the input.
Local derivative	The derivative of a single link, taken with respect to that link's own input.	One gear's ratio, ignoring the rest of the bike.
Leibniz notation	Writing a derivative as dy/dx, read "change in y per change in x".	The slope written like a fraction of tiny changes.
Chain rule	If y = f(u) and u = g(x), then dy/dx = (dy/du) × (du/dx).	Multiply the slopes of the links.
Partial derivative	The derivative with respect to one variable while all others are held fixed (see 1.06).	The slope along one direction only.
Gradient	The vector of all partial derivatives of a function (see 1.06).	An arrow of slopes, one per input.
Computational graph	A diagram where nodes are operations or values and arrows show which value feeds which.	A flowchart of the calculation.
Forward pass	Evaluating a network from input to output to get predictions and loss.	Running the chain left to right.
Backward pass	Computing derivatives of the loss from the output back to every parameter using the chain rule.	Running the slopes right to left.
Backpropagation	The algorithm that applies the chain rule efficiently over a computational graph to get all gradients.	The chain rule, organised and reused.
Loss	A single number measuring how wrong the network is.	The score we want to push down.
Activation function	A non-linear function (sigmoid, ReLU, and so on) applied inside a layer.	A bend in the chain that lets networks learn curves.
Vanishing gradient	Gradients that shrink towards zero because many small slopes are multiplied.	The signal fades before reaching early layers.
Exploding gradient	Gradients that grow enormous because many large slopes are multiplied.	The signal blows up on the way back.

4. Step-by-step explanation

Step 1: Start with a plain function and its slope

A function like y = u² takes u and returns its square. Its slope tells us how much y changes when u changes by a very small amount. For u², the slope is 2u. At u = 4 the slope is 8: a tiny nudge of 0.01 in u moves y by about 0.08.

Step 2: Connect two functions in a row

Now let u itself depend on x through u = 3x + 1. Whenever x changes, u changes, and then y changes because u changed. The variable x never touches y directly. It only reaches y through u. That is what "composite" means: y = f(g(x)) = (3x + 1)².

Step 3: Find each link's slope separately

The inner link u = 3x + 1 is a straight line with slope 3 everywhere. So du/dx = 3. The outer link y = u² has slope dy/du = 2u. At x = 1 we have u = 3 × 1 + 1 = 4, so dy/du = 2 × 4 = 8.

Step 4: Multiply

If x increases by a tiny amount, u moves 3 times as much. Then y moves 8 times as much as u moved. Chain the two amplifications: 3 × 8 = 24. So dy/dx = (dy/du) × (du/dx) = 8 × 3 = 24.

Why multiply and not add? Because each link scales the previous link's change. If one gear doubles a movement and the next gear triples it, the result is 6 times, not 5. Effects that scale each other multiply.

Step 5: Check with algebra

Expand the whole thing: y = (3x + 1)² = 9x² + 6x + 1. The derivative is 18x + 6. At x = 1 that is 24. The chain rule agrees with brute force. The chain rule matters because for complicated functions, expanding is impractical or impossible, while multiplying link slopes is always easy.

Step 6: Check with a nudge

Move x from 1 to 1.01. Then u = 3 × 1.01 + 1 = 4.03 and y = 4.03² = 16.2409. The rise in y is 16.2409 − 16 = 0.2409. The chain rule predicted 24 × 0.01 = 0.24. The tiny gap of 0.0009 appears because the slope itself changes slightly as we move; the smaller the nudge, the smaller the gap.

Step 7: Write it in general

For y = f(g(x)), the chain rule reads: the derivative is f'(g(x)) × g'(x). In words: take the derivative of the outer function, evaluated at the inner function's value, and multiply by the derivative of the inner function. A handy memory phrase is "derivative of the outside, left alone on the inside, times derivative of the inside".

Step 8: Extend to many links

If y depends on v, v depends on u, and u depends on x, then dy/dx = (dy/dv) × (dv/du) × (du/dx). Add as many links as you like; just multiply one more factor per link. A deep neural network is exactly this: dozens or hundreds of links.

5. Everything the video skipped

Notation: three ways to write the same thing

Leibniz notation writes dy/dx = (dy/du)(du/dx). It looks as if the du terms cancel, like fractions. They are not literally fractions, but the notation is designed so that the "cancelling" gives the correct answer, which makes it excellent for remembering the rule. Prime notation writes (f∘g)'(x) = f'(g(x)) · g'(x). Operator notation writes d/dx f(g(x)).

More single-variable examples

Function	Outer / inner	Derivative
(3x + 1)²	outer u², inner 3x + 1	2(3x + 1) × 3 = 6(3x + 1)
sin(x²)	outer sin(u), inner x²	cos(x²) × 2x
e^(3x)	outer e^u, inner 3x	e^(3x) × 3 = 3e^(3x)
(2x + 5)^10	outer u^10, inner 2x + 5	10(2x + 5)^9 × 2 = 20(2x + 5)^9
ln(x² + 1)	outer ln(u), inner x² + 1	2x / (x² + 1)

Chains of three or more links

For y = (sin(x²))³ there are three links: x → x² → sin → cube. Multiply the three local slopes: 3(sin(x²))² × cos(x²) × 2x. Work from the outside inwards, one layer at a time.

The multivariable chain rule: sum over paths

When a value can reach the output by more than one route, you multiply along each route and then add the routes together. If z = f(x, y), and both x and y depend on t, then dz/dt = (∂z/∂x)(dx/dt) + (∂z/∂y)(dy/dt). Multiply along a path, add across paths. This is the single most important sentence for understanding backpropagation, because one weight often influences the loss through many paths.

Computational graphs

Any calculation can be drawn as a graph. Each node does a small operation (add, multiply, sigmoid, square). Each arrow carries a value forward. On the way back, each node knows only its own local slope, and the chain rule stitches those slopes together. This is how libraries such as PyTorch and TensorFlow compute gradients automatically (called automatic differentiation, or autodiff, in reverse mode).

Forward pass versus backward pass

The forward pass computes values from input to loss. These values must be stored, because the local slopes depend on them. The backward pass starts at the loss with the number 1 (the slope of the loss with respect to itself) and moves backward. At each node it multiplies the incoming slope by that node's local slope and passes the result on. Reverse mode is efficient because one backward sweep gives the gradient for all parameters at once, instead of one sweep per parameter.

Slopes of common activation functions

The sigmoid σ(z) = 1/(1 + e^(−z)) has slope σ(z) × (1 − σ(z)), which never exceeds 0.25 (it peaks at 0.25 when z = 0). ReLU, max(0, z), has slope 1 for positive z and 0 for negative z (at exactly 0 the slope is undefined, and libraries just pick 0 or 1). The tanh function has a maximum slope of 1.

Vanishing and exploding gradients

Because the backward pass multiplies many slopes, the product can drift. Ten sigmoid layers each contributing at most 0.25 give at most 0.25^10 ≈ 0.00000095, essentially zero (this is the best case; typical slopes are smaller). Early layers then barely learn. Conversely, ten layers each contributing 1.5 give 1.5^10 ≈ 57.7, and fifty layers would give a number in the hundreds of millions. Remedies include ReLU-style activations, careful weight initialisation, normalisation layers, residual connections (shortcuts that add a path with slope 1), and gradient clipping.

History and background

Gottfried Leibniz (1670s) introduced dy/dx notation, partly because it makes the chain rule look natural. The rule was in use from the earliest days of calculus. Its famous modern application, backpropagation, was popularised for neural networks by Rumelhart, Hinton and Williams in 1986, although similar ideas had appeared earlier in control theory and in Seppo Linnainmaa's 1970 work on reverse-mode automatic differentiation.

Limitations and trade-offs

The chain rule needs each link to be differentiable. Functions with sharp corners (like ReLU at 0) need a convention. Storing every forward value for the backward pass uses a lot of memory, so some systems recompute values instead (gradient checkpointing). And a correct gradient is only a local direction: it says which way is downhill right here, not where the global minimum lies.

Where this sits on the roadmap

It builds on 1.05 (functions and slope), which gave us the single-link slope, and 1.06 (partial derivatives and the gradient), which gave us slopes for many inputs. It is what lets gradient descent work in real networks. It leads next to 1.08 (probability basics), where the course shifts from how a model learns to how it expresses uncertainty.

6. Worked examples

Example 1: Three links, with a check

Let u = 2x, then v = u + 1, then y = v². Find dy/dx at x = 3.

Forward values: u = 6, v = 7, y = 49.
Local slopes: du/dx = 2, dv/du = 1, dy/dv = 2v = 14.
Multiply: 14 × 1 × 2 = 28.
Check: y = (2x + 1)², so dy/dx = 2(2x + 1) × 2 = 4(2x + 1) = 4 × 7 = 28. They match.

Example 2: A tiny two-layer network

Setup (all numbers illustrative): input x = 1, first weight w1 = 2, second weight w2 = 3, no activation function, target = 4. The network computes h = w1 × x and output = w2 × h. The loss is L = (output − target)².

Forward pass:

h = 2 × 1 = 2.
output = 3 × 2 = 6.
L = (6 − 4)² = 4.

Backward pass, link by link:

dL/d(output) = 2 × (output − target) = 2 × 2 = 4.
d(output)/dw2 = h = 2, so dL/dw2 = 4 × 2 = 8.
d(output)/dh = w2 = 3, so dL/dh = 4 × 3 = 12.
dh/dw1 = x = 1, so dL/dw1 = 12 × 1 = 12.

Check by algebra: L = (w1 w2 x − 4)². Then dL/dw1 = 2(w1 w2 x − 4) × w2 x = 2 × 2 × 3 = 12, and dL/dw2 = 2 × 2 × (w1 x) = 4 × 2 = 8. Both match.

Gradient descent step with learning rate 0.01: w1 becomes 2 − 0.01 × 12 = 1.88, and w2 becomes 3 − 0.01 × 8 = 2.92. The new output is 1.88 × 2.92 = 5.4896, and the new loss is (5.4896 − 4)² ≈ 2.219. The loss dropped from 4 to about 2.22 in one step. With a much larger learning rate such as 0.1, the same gradients would give w1 = 0.8, w2 = 2.2, output 1.76, loss 5.02, which is worse: an overshoot. Gradients give direction, and the step size still has to be chosen sensibly.

Example 3: One neuron with a sigmoid

Input x = 1, weight w = 2, bias b = −1, target 1 (illustrative). Then z = w x + b = 1, activation a = σ(1) ≈ 0.7311, and loss L = (a − 1)² ≈ 0.0723.

dL/da = 2(a − 1) = 2 × (−0.2689) ≈ −0.538.
da/dz = a(1 − a) = 0.7311 × 0.2689 ≈ 0.1966.
dL/dz = −0.538 × 0.1966 ≈ −0.106.
dz/dw = x = 1, so dL/dw ≈ −0.106. dz/db = 1, so dL/db ≈ −0.106.

The negative signs tell us increasing w or b would lower the loss. Notice the sigmoid slope 0.1966 shrank the signal by about 80 percent in a single layer.

Example 4: Two paths added

Let z = x² + 3y, with x = t and y = t². At t = 2: x = 2, y = 4. Path through x: (∂z/∂x)(dx/dt) = 2x × 1 = 4. Path through y: (∂z/∂y)(dy/dt) = 3 × 2t = 12. Add the paths: 16. Check: z = t² + 3t² = 4t², so dz/dt = 8t = 16. Correct.

7. Comparison tables and diagrams

The chain as a picture:

x ──(slope 3)──> u ──(slope 8 at u = 4)──> y

total slope from x to y = 3 × 8 = 24

Situation	What to do with the slopes
Links in a row	Multiply
Separate routes from input to output	Multiply along each route, then add the routes
Layer in a network	Local slope times the gradient arriving from the layer above

Activation	Maximum slope	Effect on deep stacks
Sigmoid	0.25	Strong vanishing gradient risk
Tanh	1	Better, but still shrinks away from zero
ReLU	1 (for positive inputs)	Passes gradient unchanged when active; zero when inactive

Forward pass	Backward pass
Input to loss	Loss to parameters
Computes values	Computes slopes
Stores intermediate values	Uses the stored values

8. Common mistakes and myths

Forgetting the inner derivative. The derivative of (3x + 1)² is not 2(3x + 1). That misses the factor 3. The correct answer is 2(3x + 1) × 3.
Adding instead of multiplying along a chain. Links in a row multiply. Only separate paths add.
Evaluating the outer slope at the wrong point. dy/du must be evaluated at u (here 4), not at x (here 1). Using 2x = 2 instead of 2u = 8 would give 6, not 24.
Believing the chain rule is only for neural networks. It is ordinary calculus; networks simply make heavy use of it.
Thinking du and dx truly cancel like fractions. The notation helps memory, but the rule is a theorem, not cancellation.
Assuming a bigger gradient is always better. Huge gradients explode training; tiny ones stall it.
Thinking backpropagation is a different maths from the chain rule. It is the chain rule, organised so shared work is reused.
Confusing the nudge check with exactness. The nudge check is approximate; the gap shrinks as the nudge shrinks.

9. Real-life and AI/ML uses

Physics and engineering use it for rates in linked systems: if the radius of a balloon grows at some rate and volume depends on radius, volume's rate is the product of the two rates. Economics uses it when profit depends on price and price depends on time.

In AI the uses are everywhere. Training an image classifier means computing the gradient of the loss through convolution layers, activations, and a final softmax. Training a language model means chaining through attention blocks across many layers. Each parameter's gradient arrives by multiplying local slopes backward from the loss. Automatic differentiation tools implement this so researchers never write the derivative by hand. Residual connections in modern networks were designed specifically to keep the backward product healthy, and techniques such as gradient clipping exist because of the exploding-product problem.

10. Self-check questions

State the chain rule in Leibniz notation. Answer: dy/dx = (dy/du) × (du/dx).
For u = 3x + 1 and y = u², what are du/dx and dy/du at x = 1? Answer: du/dx = 3, and dy/du = 2u = 8 (because u = 4).
What is dy/dx there? Answer: 8 × 3 = 24.
Why multiply rather than add the link slopes? Answer: each link scales the change coming from the previous link, and scalings compound multiplicatively.
Differentiate e^(3x). Answer: 3e^(3x).
Differentiate (2x + 5)^10. Answer: 20(2x + 5)^9.
Differentiate sin(x²). Answer: 2x cos(x²).
In the tiny network with w1 = 2, w2 = 3, x = 1, target 4, what is dL/dw1? Answer: 12.
Why do deep sigmoid networks suffer vanishing gradients? Answer: the sigmoid slope is at most 0.25, and the backward pass multiplies one such slope per layer, so the product shrinks rapidly (0.25^10 is about 0.00000095).
When do you add in the multivariable chain rule? Answer: when a variable influences the output along several separate paths: multiply along each path, then add across paths.

11. Key takeaways, go deeper, and what is next

Key takeaways:

A composite function is a chain; its slope is the product of its links' slopes.
Evaluate each local slope at the right point, and never forget the inner derivative.
Several routes add; several links in a row multiply.
Backpropagation is the chain rule applied from the loss backwards over a computational graph.
Long products of slopes can vanish or explode, which shapes how modern networks are designed.

Go deeper:

Practise differentiating five composite functions by hand and checking each with a nudge.
Draw the computational graph of a one-neuron network and label every local slope.
Read about automatic differentiation (forward mode versus reverse mode).
Look up residual connections and why they help gradients.
Revisit 1.05 and 1.06 to see how single-link slopes and gradients combine here.

Next on the roadmap: 1.08 Probability basics.

Learn AI by Story: one concept, one story, every day.

The chain rule: study notes