1.16 Landscapes: convex, local minima, saddle points: study notes

Learn AI by Story

1. The idea in one sentence

Training a neural network means walking downhill on a landscape made of the loss, and the shape of that landscape (one bowl, many valleys, passes and flat plains) decides whether simple downhill steps find a good answer.

This is concept 1.16 on the Learn AI by Story roadmap. It sits between the linear algebra of 1.15 (eigenvectors and SVD) and the next idea, 1.17 (vector norms), and it leans on calculus ideas from 1.07 (the chain rule).

2. The story from the video

Lena is lost on a hilly landscape in thick fog and can feel only the slope under her feet. Training a network is the same: the landscape is the loss, and the goal is the lowest point.

The friendly case is a convex bowl, y = x², with exactly one bottom. Real networks are not convex. The video shows a curve with two valleys, one dipping to −1.51 and one only to −0.52, and gradient descent from x = −1.5 versus x = +1.5 ends in different valleys.

Then it adds the saddle point of f = x² − y², where the slope is zero yet the point is a minimum along x and a maximum along y. Near it, x shrinks by a factor of 0.8 per step while y grows by 1.2, so descent escapes after a while.

The video ends with the practitioner view: in large networks saddles outnumber bad traps, most valleys are already low, and tricks such as random starts, momentum and mini-batch noise help. The real troublemakers are wide flat plateaus.

3. Key definitions

Each entry gives a precise definition, then the plain-English meaning.

Loss function (cost). A number that measures how wrong the model's predictions are on the training data, as a function of the parameters. Plain English: the height of the ground we are standing on.
Parameters (weights). The adjustable numbers in the model. Plain English: the map coordinates of our position; with a million weights the landscape has a million horizontal directions.
Loss landscape. The graph of the loss plotted against the parameters. Plain English: the terrain; low is good.
Gradient. The vector of partial derivatives of the loss with respect to each parameter; it points in the direction of steepest increase. Plain English: the uphill arrow under your feet.
Gradient descent. Repeatedly updating the parameters by subtracting a small multiple of the gradient: new = old − (learning rate) × gradient. Plain English: always step downhill, a little at a time.
Learning rate. The step-size multiplier in gradient descent. Plain English: how long each stride is.
Critical (stationary) point. Any point where the gradient is zero. Plain English: the ground is locally level; it could be a bottom, a top or a pass.
Local minimum. A point where the loss is no larger than at every nearby point. Plain English: the bottom of some valley, not necessarily the deepest one.
Global minimum. A point where the loss is no larger than anywhere at all. Plain English: the bottom of the deepest valley.
Local maximum. The mirror image: no smaller than all nearby points. Plain English: a hilltop.
Saddle point. A critical point that is a minimum along some directions and a maximum along others. Plain English: a mountain pass, or the middle of a horse saddle.
Convex set. A set where the straight segment between any two of its points stays inside the set. Plain English: no dents.
Convex function. A function f where, for any two points x and x' and any weight t between 0 and 1, t·f(x) + (1 − t)·f(x') ≥ f(t·x + (1 − t)·x'). Plain English: the straight line (chord) between two points on the graph never dips below the graph, so the graph is bowl-shaped.
Hessian. The square matrix of all second derivatives of the loss. Plain English: it describes how the ground curves in every direction.
Positive semidefinite (PSD). A symmetric matrix H with vT H v ≥ 0 for every vector v; equivalently, all eigenvalues are 0 or positive (see 1.15). Plain English: curving upward or flat in every direction, never downward.
Positive definite. Same, but strictly greater than zero for every nonzero v. Plain English: curving strictly upward in every direction.
Eigenvalue of the Hessian. The curvature along one special direction (an eigenvector). Plain English: a positive one means a bowl direction, a negative one means a hill direction.
Plateau. A wide region where the gradient is near zero. Plain English: a flat plain where you barely move.
Ravine (ill-conditioning). A landscape where curvature is steep in some directions and shallow in others. Plain English: a long narrow canyon.
Condition number. The ratio of the largest to the smallest Hessian eigenvalue. Plain English: how stretched the canyon is.
Symmetry breaking. Giving identical units different starting values so they can learn different things. Plain English: do not start every unit as a clone.
Non-identifiability. Different parameter settings that produce exactly the same function. Plain English: many different addresses lead to the same house.

4. Step-by-step explanation

Step 1: Turn learning into a hill-walking problem. A model has parameters. For every choice of parameters there is a loss. Plot loss against parameters and you get a landscape. Training searches that landscape for low ground. With one parameter the landscape is a curve; with two it is a surface you can picture; with a million it is a surface we cannot picture but can still compute on.

Step 2: The only tool is the local slope. Like Lena in the fog, gradient descent sees only the gradient at the current point. It cannot see the whole map. The update is:

\theta_{new} = \theta_{old} - \eta \, \nabla L(\theta_{old})

Here theta is the parameter vector, eta is the learning rate and the upside-down triangle (nabla L) is the gradient of the loss. Computing the gradient for a network uses the chain rule from 1.07 (this is backpropagation). Averages over data use the sampling ideas of 1.12.

Step 3: The best case, a convex bowl. Take L(x) = x². The slope is 2x, so at x = 3 the slope is 6, pointing uphill; stepping against it moves us toward 0. There is exactly one place where the slope is zero, x = 0, and it is the lowest point everywhere. A function is convex when it has this bowl shape, formally when its chord never dips below its graph. For a smooth function there is an easy test: the second derivative (in many variables, the Hessian) must be non-negative (PSD) everywhere. For x² the second derivative is 2, which is positive. Consequence: in a convex landscape every local minimum is the global minimum, so any method that goes downhill and settles has found the best answer. A convex function can still have a flat bottom or no bottom at all (e^x keeps falling toward minus infinity), but it cannot have a trap.

Step 4: Why that guarantee is lost. Neural-network losses are not convex in their parameters. One simple reason: hidden units can be swapped around without changing the network's function, so the landscape contains several identical copies of any valley, and mixing two good solutions does not generally give a good solution. Convexity requires the average of two points to have loss no higher than the average of their losses, and that fails. The video's curve f(x) = x⁴ − 2x² + 0.5x shows it: the two valley bottoms have loss −1.515 and −0.517, but halfway between them (x = −0.064) the loss is about −0.04, which is higher than the average of the two (about −1.016).

Step 5: Local minima and the starting point. With two valleys, where you start decides where you finish. Gradient descent with learning rate 0.05 for 200 steps ends at x = −1.057 (loss −1.515) from a start of −1.5, and at x = 0.930 (loss −0.517) from a start of +1.5. Same rule, same data, loss differing by 1.0. The shallow valley is a local minimum; the deep one is the global minimum. At both bottoms the slope is zero and the second derivative is positive (about 6.4 at the shallow one, 9.4 at the deep one).

Step 6: Higher dimensions bring saddle points. In two or more directions a zero-slope point can curve up along one direction and down along another. The standard example is f(x, y) = x² − y². At the origin the gradient is (0, 0). The Hessian is diag(2, −2), with eigenvalues +2 and −2. Mixed signs mean saddle. If all eigenvalues are positive the critical point is a local minimum, if all are negative a local maximum, and if they are mixed it is a saddle.

Step 7: Can descent escape a saddle? Gradient descent with rate 0.1 gives x_new = x − 0.1·2x = 0.8x and y_new = y + 0.1·2y = 1.2y. The x part collapses toward zero, while the y part grows geometrically. If y is exactly zero it never grows and descent stalls on the pass; in practice noise or rounding makes y tiny but nonzero, and 1.2 to the power 25 is about 95, so a tiny y grows about a hundredfold in 25 steps. Then the walker slides off the pass and keeps descending. The cost is time: near a saddle the gradient is small, so progress is slow.

Step 8: What the evidence says about big networks. The Deep Learning book (Goodfellow, Bengio and Courville, chapter 8) reports that, in high dimensions, the expected ratio of saddle points to local minima grows exponentially with the number of dimensions n, and that experts suspect that for large enough networks most local minima have a low cost. It also says it is still an open question whether practical networks have many local minima of high cost, and recommends plotting the gradient norm over time: if it does not shrink to an insignificant size, the problem is not a local minimum or other critical point at all.

Step 9: The real difficulties and the helpers. Wide flat regions (plateaus), where the gradient and Hessian are zero, are named in the same chapter as a major problem for numerical optimizers. Narrow ravines slow descent. The standard helpers are random initialization (breaks symmetry between identical units), momentum (carries speed across flat patches), and mini-batch noise (jiggles the walker off sharp saddles). The mini-batch and momentum remarks are the commonly described practitioner view rather than strict guarantees.

5. Everything the video skipped

The video is nine short scenes, so several important ideas were left out.

The second-derivative test in many dimensions. At a critical point, look at the eigenvalues of the Hessian (the eigenvalue idea from 1.15 returns here, and for a symmetric matrix like the Hessian they are all real). All positive: local minimum. All negative: local maximum. Mixed signs: saddle. Some zero: the test is inconclusive and you need more information. This is why saddles are described as having both upward and downward directions: the signs of the eigenvalues are the directions.

Why saddles become common in high dimensions (a rough intuition). Imagine a critical point in a space with n directions. For it to be a true local minimum, every one of the n curvatures must be positive. If each curvature behaved like a coin flip, the chance of all n coming up positive would shrink quickly as n grows, while mixed signs become the usual outcome. This is a heuristic, not a proof; the Deep Learning book states the stronger result as holding for certain classes of functions, and notes that how well it describes real networks is still being studied.

Plateaus and dead units. A plateau is a region where the gradient is almost zero. Examples noted in the Toronto CSC421 lecture notes include saturated units (a sigmoid pushed far into its flat tail has a tiny derivative, so its weights barely update) and dead ReLU units (a ReLU that always receives a negative input outputs zero and gets exactly zero gradient). Plateaus have no direction to escape in, which is why researchers fear them more than saddles.

Ravines and ill-conditioning. When the Hessian has very large and very small eigenvalues, gradient descent oscillates across the steep direction and crawls along the shallow one. The learning rate must stay below 2 divided by the largest eigenvalue to avoid diverging, so progress along the flat direction is governed by the condition number (largest over smallest eigenvalue). Remedies include normalising inputs, momentum, and adaptive methods such as RMSprop and Adam (named in the Toronto notes).

Symmetry and zero initialisation. If all weights start at zero (or identical hidden units share identical weights), those units receive identical updates forever, so they never differ. Random initialisation breaks the symmetry. That is the "random start" tip in the video.

Non-identifiability. Swapping two hidden units gives a different parameter vector with exactly the same function. So a trained network sits in one of many equivalent valleys. Many local minima therefore exist, but many of them are copies of each other and equally good, which is one reason their sheer number is not alarming.

Which common models are convex. Squared-error linear regression, logistic regression and softmax regression have losses that are convex in their parameters, so descent has a single basin to find. Hidden layers with nonlinear activations destroy this. Convexity is also what makes classic tools such as support vector machines reliably solvable.

Jensen's inequality. For a convex function f, the average of f over several points is at least f of the average: E[f(X)] ≥ f(E[X]) (d2l.ai, convexity chapter). It is the same chord-above-graph idea, and it connects to the probability of 1.08 to 1.11 (expectations, variance) and to the bounds used in Bayesian methods (1.09).

Beyond plain gradient descent. Second-order methods use the Hessian directly, which tells them about curvature, but a full Hessian for a million parameters has a trillion entries, so it is rarely used raw. Practical optimisers (momentum, Adam) approximate curvature cheaply. Stochastic gradient descent adds sampling noise (1.12 on sampling: a mini-batch gradient is an estimate of the full gradient), and that noise is commonly described as helping to leave saddles and sharp valleys.

Limitations and trade-offs. Reaching a minimum of the training loss is not the same as generalising well; a low training loss in a deep valley can still overfit. The loss landscape also changes with the data, the architecture and the loss itself, so statements such as "most local minima are good" are beliefs about typical large networks, not laws. Pictures of landscapes in papers are always low-dimensional slices of a space with millions of dimensions, so they can mislead.

Looking ahead. Concept 1.17 covers vector norms, which measure the length of a gradient or a weight vector. The gradient norm is exactly the diagnostic the Deep Learning book recommends for telling a true critical point from a plateau or other difficulty.

6. Worked examples

All numbers below were checked with a short Python calculation. Where a number depends on my choice of start or step size, it is marked illustrative.

Example A: two valleys, two destinations

Loss: f(x) = x⁴ − 2x² + 0.5x, so the slope is f'(x) = 4x³ − 4x + 0.5. Learning rate 0.05.

Start at x = +1.5:

f(1.5) = 5.0625 − 4.5 + 0.75 = 1.3125. Slope = 4(3.375) − 6 + 0.5 = 8.0.
Step 1: x = 1.5 − 0.05 × 8.0 = 1.1. Loss −0.406, slope 1.424.
Step 2: x = 1.1 − 0.05 × 1.424 = 1.029. Loss −0.482, slope 0.740.
Step 3: x = 0.992, loss −0.504. After 200 steps: x = 0.930, loss −0.517, slope essentially 0. Stuck in the shallow valley.

Start at x = −1.5:

f(−1.5) = −0.1875. Slope = 4(−3.375) + 6 + 0.5 = −7.0.
Step 1: x = −1.5 − 0.05 × (−7.0) = −1.15. Loss −1.471, slope −0.984.
Step 2: x = −1.101, loss −1.506. Step 3: x = −1.079, loss −1.513.
After 200 steps: x = −1.057, loss −1.515. Deep valley.

Result: same rule, different start, final losses −0.517 versus −1.515, a gap of about 1.0. The third zero-slope point, near x = 0.127, is a local maximum (second derivative about −3.8), the hilltop between the two valleys. The valleys have second derivatives of about +6.4 (shallow) and +9.4 (deep).

Example B: classifying critical points with the Hessian

For each function the gradient is zero at the origin; the Hessian tells the type.

Function	Hessian	Eigenvalues	Verdict
x² + y²	diag(2, 2)	2, 2	local (and global) minimum
x² − y²	diag(2, −2)	2, −2	saddle
x² + xy + y²	[[2, 1], [1, 2]]	1, 3	local minimum
x² + 4xy + y²	[[2, 4], [4, 2]]	−2, 6	saddle

The last one is the instructive case: both squared terms are positive, yet the cross term 4xy is strong enough to give a negative eigenvalue, so the origin is a saddle. You cannot judge by looking at the diagonal alone.

Example C: escaping a saddle step by step

f(x, y) = x² − y², learning rate 0.1. Gradient = (2x, −2y), so each step multiplies x by 1 − 0.2 = 0.8 and y by 1 + 0.2 = 1.2. Start at (1, 0.01).

Step	x	y
0	1.000	0.0100
1	0.800	0.0120
5	0.328	0.0249
10	0.107	0.0619
25	0.0038	0.954

After 25 steps y has grown by 1.2 to the power 25, about 95.4 times, from 0.01 to about 0.95. Solving 1.2 to the power n = 100 gives n of about 25.3 steps for a hundredfold increase. Compare a start of y = 0.0001 exactly at the same x: it needs 51 steps to reach |y| of 1 (illustrative). A start exactly at y = 0 never leaves: after 50 steps x is 0.000014 and y is still 0, so the walker has converged to the saddle. This is why noise helps.

Example D: a ravine and the 2 over lambda rule

Take f(x, y) = 0.5 (x² + 10 y²). The Hessian is diag(1, 10), a condition number of 10. Gradient descent multiplies x by (1 − η) and y by (1 − 10η).

With η = 0.15 (below 2/10 = 0.2): x factor 0.85, y factor −0.5. From (1, 1) the steps give (0.85, −0.5), (0.7225, 0.25), (0.614, −0.125). The y coordinate flips sign each step while shrinking, the zig-zag across the steep wall; x creeps down slowly.
With η = 0.05: y factor 0.5, x factor 0.95. After 5 steps x is still 0.774 while y is 0.031. The flat direction is the bottleneck.
With η = 0.25 (above 0.2): y factor −1.5, so |y| grows by 1.5 each step and after 5 steps is about 7.6 times larger: divergence.

This is the trade-off: the steep direction caps the learning rate, and the flat direction then sets how long training takes.

7. Comparison tables and diagrams

Table 1: Types of zero-gradient points.

Point type	Gradient	Hessian eigenvalues	Shape	Gradient descent behaviour
Local/global minimum	0	all positive	bowl	settles and stays
Local maximum	0	all negative	hilltop	pushed away; unstable
Saddle point	0	mixed signs	mountain pass	can stall, small noise lets it escape
Plateau	near 0	near 0 (flat)	plain	very slow, no clear direction

Table 2: Convex versus non-convex landscapes.

Property	Convex	Non-convex (neural networks)
Local minima	every one is global	many, depth can differ
Result depends on start?	no (the value reached does not)	yes
Hessian	PSD everywhere	mixed signs somewhere
Saddles	none that trap	common in high dimensions
Typical models	linear, logistic and softmax regression	multi-layer networks
Guarantee	descent reaches the best answer	none, only good practice and experience

Table 3: Landscape troubles and standard remedies.

Trouble	Symptom	Common remedy
Bad local minimum	loss stalls high, gradient is near zero	restarts from different random seeds, larger network
Saddle point	loss stalls briefly, then drops	noise (mini-batches), momentum
Plateau	loss almost flat for long periods	better activation choice (avoid saturation), momentum, adaptive optimisers
Ravine	loss zig-zags, slow progress	normalise inputs, momentum, Adam or RMSprop
Symmetry	all units in a layer learn the same thing	random initialisation

Diagram: the three landscapes side by side (text sketch).

CONVEX BOWL        TWO VALLEYS            SADDLE (side view along x, then y)

\        /         \    /\    /          along x:  \_/   (minimum)
 \      /           \  /  \  /           along y:  /^\   (maximum)
  \____/             \/    \/
 one bottom          deep  shallow
                     (global) (local)

Reading the sketch: the bowl has a single lowest point; the two-valley curve has a deep (global) and a shallow (local) bottom with a hump between; the saddle goes up in one direction and down in the other, like a pass between two peaks.

Diagram: where a descent path ends up (Example A).

Start x	Direction of first step	Valley reached	Final loss
−1.5	right (slope is negative)	deep, x = −1.057	−1.515
+1.5	left (slope is positive)	shallow, x = 0.930	−0.517

The hump near x = 0.127 acts as a watershed: starts to its left drain into the deep valley, starts to its right drain into the shallow one (for small enough steps).

8. Common mistakes and myths

Myth: neural networks get stuck in bad local minima all the time. The Deep Learning book says practitioners once blamed local minima for most difficulty but that this no longer appears to be the case; the evidence is that many minima are low and saddles and plateaus are more common troubles. Whether practical networks have many high-cost local minima is still described as open.
Mistake: a zero gradient means a minimum. It only means a critical point: minimum, maximum, saddle or plateau. Check the Hessian eigenvalues or the loss itself.
Mistake: a flat loss curve means we reached a minimum. A plateau or a saddle looks the same on a loss plot. The book suggests plotting the gradient norm: if it is not tiny, you are not at any critical point.
Myth: a saddle point stops gradient descent forever. Exactly on the saddle it can stall, but tiny perturbations make the unstable direction grow (1.2 per step in Example C), so descent slides off, only more slowly.
Mistake: positive diagonal Hessian entries imply a minimum. Example B showed x² + 4xy + y² has positive diagonal and a saddle at the origin. Use eigenvalues.
Mistake: initialising all weights to zero is a safe neutral start. Identical units receive identical updates, stay identical, and the layer collapses to one effective unit.
Myth: convex means easy and non-convex means impossible. Convex functions can have flat bottoms or no minimum at all; non-convex networks are trained successfully every day.
Mistake: a bigger learning rate always finds a better valley. Above 2 over the largest curvature, descent diverges (Example D); a large rate can also overshoot narrow valleys.
Mistake: the pictures in papers are the real landscape. They are one- or two-dimensional slices of a space with millions of dimensions.

9. Real-life uses

Training any deep network. Image classifiers, language models and speech systems are all trained by walking a non-convex loss landscape with gradient-based methods. The fact that this works reliably in practice, despite the lack of convexity, is the situation the video describes.
Choosing a convex model on purpose. When a guarantee matters, such as a quick, repeatable baseline for tabular data, practitioners pick logistic regression or linear regression, whose loss has one basin, so every run lands on the same answer regardless of start.
Random restarts. For small non-convex problems (for example clustering with k-means or small neural nets) people run training from several random starts and keep the lowest loss, a direct response to the two-valley picture in Example A.
Random initialisation of layers. Every deep learning library initialises weights randomly, not at zero, to break symmetry between hidden units.
Diagnosing a stalled run. Engineers watch the loss and the gradient norm. A flat loss with a tiny gradient points to a plateau, saddle or minimum; a flat loss with a large gradient points to an oscillating learning rate, as in Example D.
Optimiser choice. Momentum and Adam are used because real losses have ravines and plateaus, not because of bad local minima.
Activation-function choice. Using ReLU-family activations instead of saturating sigmoids reduces the flat-tail plateaus mentioned in the Toronto notes, although a ReLU can still die if it is always negative.
Fine-tuning pretrained models. Starting from a pretrained network means starting already inside a good valley, which is why small learning rates and short training often suffice (a common practitioner view).

10. Self-check questions

What makes a function convex, in plain words? Its graph is bowl-shaped: the straight line between any two points on the graph never dips below the graph.
- Answer: that is the chord definition; for smooth functions it matches a second derivative (Hessian) that is never negative.
Why is a convex loss easy to optimise? Every local minimum is a global minimum.
- Answer: so descent that settles anywhere has found the best value; the start does not change the value reached.
For f(x) = x⁴ − 2x² + 0.5x, where do the two valleys end up from starts of −1.5 and +1.5 (rate 0.05, 200 steps)?
- Answer: x = −1.057 (loss −1.515) and x = 0.930 (loss −0.517) respectively.
What is a saddle point and how does its Hessian look?
- Answer: gradient is zero, but the Hessian has both positive and negative eigenvalues. For x² − y² the eigenvalues are +2 and −2.
Classify the origin for f = x² + 4xy + y².
- Answer: Hessian [[2, 4], [4, 2]] has eigenvalues −2 and 6, so it is a saddle even though both squared terms are positive.
For gradient descent on x² − y² with rate 0.1, by what factor does y change each step, and how large is y after 25 steps if it starts at 0.01?
- Answer: factor 1.2 each step; 1.2 to the power 25 is about 95.4, so y is about 0.95.
What happens if y starts at exactly 0?
- Answer: y stays 0 forever, x shrinks to zero, and descent converges to the saddle. Any tiny noise breaks this.
Name two things other than local minima that slow neural-network training.
- Answer: plateaus (saturated sigmoids, dead ReLUs) and ravines (ill-conditioned curvature); also symmetry from identical initial weights.
Why should you never initialise all weights to zero?
- Answer: hidden units with identical weights get identical gradients and stay identical, so the layer behaves like a single unit; random initialisation breaks the symmetry.
Your loss curve is flat. How can you tell a true critical point from a plateau-like difficulty?
- Answer: plot the gradient norm over time (as the Deep Learning book suggests): if it is not tiny, you are not at a critical point.
In the ravine f = 0.5(x² + 10y²), what is the largest stable learning rate and why?
- Answer: just below 2 divided by the largest Hessian eigenvalue, 2/10 = 0.2; beyond it the y factor 1 − 10η has size greater than 1, so y diverges.

11. Key takeaways and go deeper

Key takeaways

Convex means one bottom: any local minimum is the global minimum, and linear, logistic and softmax regression losses are convex in their parameters.
Neural-network losses are not convex, so the starting point can decide which valley you reach (Example A: −1.515 versus −0.517).
A saddle point has zero gradient and mixed-sign Hessian eigenvalues; descent escapes through the downward direction because it grows geometrically (1.2 per step in Example C).
In large networks, saddles are believed to be more common than bad local minima, and many local minima are low; plateaus and ravines are the more practical troubles.
Helpers: random initialisation, momentum, mini-batch noise, adaptive optimisers, sensible activations.
Check the gradient norm before blaming a local minimum.

Go deeper

Work through the Deep Learning book chapter 8 on optimisation challenges.
Read the Toronto CSC421 lecture 7 slides on optimisation difficulties.
Re-derive Example C with a different learning rate and find the rate at which the x direction starts to diverge (hint: rate above 1).
Review 1.15 eigenvalues, because Hessian eigenvalues are exactly what classifies critical points.
Review 1.07 (the chain rule) to see how the gradient is computed in a network.
Look up second-order methods and the condition number once you are comfortable with ravines.

Next roadmap ID: 1.17 Vector norms, the tool for measuring the length of gradients and weights.

12. Sources and further reading

Deep Learning, chapter 8: Optimization for Training Deep Models, Goodfellow, Bengio and Courville (local minima, saddle points, plateaus, gradient-norm test).
CSC421 Lecture 7: Optimization, Roger Grosse, University of Toronto, 2019 (convexity and the Hessian, plateaus, ravines, symmetry and initialisation).
Dive into Deep Learning: Convexity, d2l.ai (convex sets and functions, Jensen's inequality, local equals global minimum, Hessian test).
Dive into Deep Learning: Optimization and Deep Learning, d2l.ai (local minima, saddle points, Hessian eigenvalue classification, vanishing gradients).

Accuracy note: The function values, minima positions and saddle-step numbers come from the video and were re-checked in Python. The starting points, learning rates and the ravine function in Examples C and D (other than the video's own saddle numbers) are illustrative choices of mine, as is the coin-flip intuition for why saddles become common; statements about large networks (saddles outnumber bad local minima, most minima are low) are expert beliefs reported by the sources, not proven facts. Remarks about momentum, mini-batch noise, fine-tuning, SVMs and restarts reflect common practitioner descriptions rather than something verified in the sources above.

Search This Blog