Learn AI by Story

  1.16 Landscapes: convex, local minima, saddle points: study notes Learn AI by Story 1. The idea in one sentence Training a neural network means walking downhill on a landscape made of the loss, and the shape of that landscape (one bowl, many valleys, passes and flat plains) decides whether simple downhill steps find a good answer. This is concept 1.16 on the Learn AI by Story roadmap. It sits between the linear algebra of 1.15 (eigenvectors and SVD) and the next idea, 1.17 (vector norms), and it leans on calculus ideas from 1.07 (the chain rule). 2. The story from the video Lena is lost on a hilly landscape in thick fog and can feel only the slope under her feet. Training a network is the same: the landscape is the loss, and the goal is the lowest point. The friendly case is a convex bowl, y = x², with exactly one bottom. Real networks are not convex. The video shows a curve with two valleys, one dipping to −1.51 and one only to −0.52, and gradient descent from x = −1.5 versus x ...

  1.15 Eigenvectors and SVD: study notes Learn AI by Story Concept 1.15, the directions that matter. Video story key: p115. Previous: 1.14 Cross-entropy and KL divergence. Next: 1.16 Landscapes: convex, local minima, saddle points. 1. The idea in one sentence An eigenvector of a matrix is a direction the matrix only stretches (never turns), and the singular value decomposition (SVD) extends this to every matrix of any shape by writing it as rotate, stretch, rotate, so the biggest stretches show which directions matter most. 2. The story from the video Pia presses on a rubber sheet and watches little arrows move. Most arrows turn, but a few special ones only get longer or shorter. With the matrix A = [[2, 1], [2, 3]], the arrow (1, 1) becomes (3, 5), a different direction, while (1, 2) becomes (4, 8), exactly 4 times itself. So (1, 2) is an eigenvector with eigenvalue 4; the other eigenvalue is 1 with eigenvector (1, −1). The video then moves to SVD, A = UΣVᵀ, using B = [[4, 0], [3,...

  1.14 Cross-entropy and KL divergence: study notes Learn AI by Story Concept 1.14, comparing distributions. Video story key: p114. Previous: 1.13 Entropy. Next: 1.15 Eigenvectors and SVD. 1. The idea in one sentence Cross-entropy is the average surprise (cost in bits) you pay when the world follows distribution P but you predict with distribution Q, and the KL divergence is the extra cost caused purely by Q being wrong, so training a model to lower cross-entropy is the same as pulling its beliefs toward the truth. 2. The story from the video Omar believes a coin lands heads 90 percent of the time, but in truth it is 50-50. Calling the truth P and the belief Q, the video adds up the surprise Omar suffers on real coin flips: 0.5 × 0.152 for heads plus 0.5 × 3.322 for tails, which is 1.737 bits. The truth alone costs only 1 bit (its entropy), so the wrong belief wastes 0.737 bits. That waste is the KL divergence, and the identity cross-entropy = entropy + KL ties the three numbers to...

  1.13 Entropy: measuring surprise: study notes Learn AI by Story 1. The idea in one sentence Entropy is the average surprise of a random outcome, counted in bits: it is low when the outcome is predictable, high when it is uncertain, and it is the lowest average number of yes/no answers (bits) any code needs to describe the outcome. 2. The story from the video Isha checks the weather forecast. Some news barely surprises her and some is a shock, so can surprise be a number? Rare events carry more surprise: chance 1/2 is 1 bit, 1/8 is 3 bits, 1/64 is 6 bits. Entropy is the average surprise over all outcomes: a fair coin has 1 bit, a coin with 90% heads has 0.47 bits, an always-heads coin has 0. With four weathers (sun 1/2, cloud 1/4, rain 1/8, snow 1/8) the average is 1.75 bits, and the code 0, 10, 110, 111 averages exactly 1.75 bits, which Shannon showed no code can beat. 3. Key definitions Term Precise definition Plain English Probability, p A number from 0 to 1 giving how likely a...