Logs, exponentials, sigmoid and softmax: study notes

 

1.28 Logs, exponentials, sigmoid and softmax: study notes

Learn AI by Story

1. The idea in one sentence

Exponentials turn adding into multiplying, logs undo them, and the two together give us sigmoid and softmax: smooth machines that squash any raw scores into probabilities between 0 and 1 that add up to 1.

2. The story from the video

Tara's chatbot cannot answer in probabilities; it gives raw scores, called logits, of 2, 1 and 0.1 for three possible replies. Scores can be negative or huge, so she cannot read them as chances. She raises e (about 2.718) to each score, which makes every number positive: 7.389, 2.718 and 1.105. Then she divides each by the total, 11.2125, and gets 0.659, 0.242 and 0.099. Now the three numbers are positive and add to 1, so they read as probabilities. With only two options the same trick collapses into a single S-shaped curve, the sigmoid.

3. Key definitions (glossary)

• Exponent / power: a^n means a multiplied by itself n times. Plain English: repeated multiplication.
• Exponential function: f(x) = a^x with a fixed base a and a variable exponent x. Plain English: growth where each step multiplies by the same factor.
• e (Euler's number): about 2.71828, the base for which the slope of e^x equals e^x itself. Plain English: the natural growth rate. It appears as the limit of (1 + 1/n)^n as n grows.
• exp(x): another way to write e^x. Handy when the exponent is a long expression.
• Logarithm: log_b(y) is the exponent you need to raise b to in order to get y. Plain English: 'how many times do I multiply?'. If 10^3 = 1000 then log base 10 of 1000 is 3.
• Natural log (ln or log): the logarithm with base e. In machine learning, 'log' almost always means natural log.
• Inverse functions: log(exp(x)) = x and exp(log(y)) = y for y above 0. Plain English: each undoes the other.
• Product rule for logs: log(ab) = log a + log b. Plain English: logs turn multiplying into adding.
• Power rule for logs: log(a^n) = n log a. Plain English: an exponent drops down to the front.
• Quotient rule: log(a/b) = log a − log b.
• Change of base: log_b(y) = ln y divided by ln b.
• Log of 1 and log of 0: log 1 = 0 for any base, and log of a number approaching 0 goes to minus infinity. Log of a negative number is not defined for real numbers.
• Logit (raw score): an unnormalised score that can be any real number, positive or negative. Also, in statistics, the logit function is log(p/(1−p)).
• Odds: p/(1−p), the chance an event happens divided by the chance it does not. Log-odds is the log of the odds, which equals the logit function.
• Sigmoid (logistic function): σ(x) = 1 / (1 + exp(−x)). Plain English: an S-shaped curve that maps any real number into the range between 0 and 1, with σ(0) = 0.5.
• Saturation: the flat regions of the sigmoid at large positive or negative inputs, where the output barely changes and the gradient is almost zero.
• Softmax: softmax(z)_i = exp(z_i) divided by the sum over j of exp(z_j). Plain English: exponentiate every score then divide by the total so the results sum to 1.
• Softplus: ζ(x) = log(1 + exp(x)). Plain English: a smooth version of the ReLU function max(0, x).
• Probability vector: a list of non-negative numbers that sums to 1.
• Temperature: a number T used as softmax(z / T). Low T sharpens the distribution toward the top score; high T flattens it.
• Cross-entropy / negative log-likelihood: the loss −log of the probability given to the correct answer. Plain English: it punishes confident wrong answers heavily.
• Log-sum-exp (LSE): log of the sum of exp of the scores; the normaliser that appears when softmax is written in log form. It returns in 1.29.
• Overflow and underflow: when exp of a big number is too large for the computer (it becomes infinity) or exp of a very negative number rounds to zero.

4. Step-by-step explanation, from basics to the full picture

Step 1. Exponentials: growth by multiplying. Writing 2^3 = 8 means three doublings. The key property is that exponents add when you multiply: 2^3 x 2^4 = 2^7. That rule is the heart of everything that follows. The special base e is chosen because it makes calculus tidy: the slope of e^x at any point is e^x. So exp grows at a rate equal to its own size. Useful values: exp(0) = 1, exp(2) = 7.389, exp(5) = 148.41. Growth is quick. Also exp of a negative number is a small positive fraction, and exp is never zero or negative for real inputs.

Step 2. Logs: the undo button. A log asks the opposite question: to what power must I raise the base to get this number? Since e^3 is about 20.09, the natural log of e^3 is exactly 3. Because exp turns adding into multiplying, log turns multiplying into adding: log10(100) + log10(1000) = 2 + 3 = 5 = log10(100000). This is why logs were used to build slide rules and why machines use them to turn long products of tiny probabilities into manageable sums.

Step 3. Why logs matter in AI. The probability of a whole dataset is a product of many per-example probabilities, often tiny. A product of a thousand numbers below 1 is practically zero for a computer. Take logs and the product becomes a sum, which is stable and which gradient descent handles well. Maximising the likelihood is the same as maximising the log-likelihood because log only stretches the scale without changing which input wins (it is increasing).

Step 4. The sigmoid: squash one score into a probability. Given any real score x, sigmoid computes 1 / (1 + exp(−x)). When x = 0, exp(0) = 1 and the result is 1/2. When x is large and positive, exp(−x) is almost 0 and the output approaches 1. When x is large and negative, exp(−x) is huge and the output approaches 0. The result sits strictly between 0 and 1, so it can be read as a probability of 'yes'. The symmetry is neat: 1 − σ(x) = σ(−x). So σ(2) = 0.8808 and σ(−2) = 0.1192, which add to 1.

Step 5. The sigmoid's slope. The derivative is σ'(x) = σ(x)(1 − σ(x)). It is largest at x = 0, where it is 0.5 x 0.5 = 0.25, and it shrinks toward zero on both sides. That shrinking is called saturation. It makes learning slow for neurons that are very confident, and it is one reason deep networks with many stacked sigmoids suffered from vanishing gradients, a reason ReLU became popular in hidden layers. Sigmoid still shines at the output of a binary classifier, where the loss function cancels much of the flatness.

Step 6. The softmax: squash many scores into a probability vector. Softmax does three jobs in two moves. Exponentiating makes every score positive, keeps the ranking (bigger score gives bigger result) and exaggerates the gaps. Dividing by the total makes the results sum to 1. For scores 2, 1, 0.1, the exponentials are 7.389, 2.718 and 1.105, the sum is 11.2125, and the probabilities are 0.659, 0.242 and 0.099. The top score gets the biggest share, but the others are not crushed to zero, hence the name 'soft' max.

Step 7. Shift invariance. Add the same constant c to every score and softmax does not change: softmax(z + c) = softmax(z). The reason is that exp(z + c) = exp(c) x exp(z), so exp(c) appears in both the top and the bottom and cancels. Only differences between scores matter. This is the key to the stability trick in 1.29: subtract the biggest score before exponentiating.

Step 8. Softmax with two classes is a sigmoid. With scores a and b, softmax gives the first class exp(a) / (exp(a) + exp(b)). Divide top and bottom by exp(b) and you get 1 / (1 + exp(−(a − b))) = σ(a − b). A check with a = 1.5 and b = 0.2 gives 0.7858 both ways. So sigmoid is the two-class special case of softmax, and everything that is true of one is true of the other.

Step 9. From scores to learning: the loss. To train, we compare the predicted probabilities with the right answer using the negative log of the probability assigned to the correct class. A model that gives the right answer 0.99 pays almost nothing, and one that gives it 0.01 pays a lot. Combining softmax with this log is the cross-entropy loss, and the log cancels the exp, leaving a clean gradient: predicted probability minus the true label.

5. Everything the video skipped

Formulas in simple words.

• Softmax: exp each score, divide by the total of all the exps.
• Sigmoid: one divided by (one plus exp of minus x).
• Derivative of sigmoid: sigmoid times (one minus sigmoid).
• Logit function, the inverse of sigmoid: log of p divided by (1 minus p). Plug in p = 0.8808 and you get back 2.0000 (checked numerically).
• Softplus: log of (one plus exp of x); its slope is the sigmoid, and softplus(x) minus softplus(minus x) equals x (Deep Learning book, section 3.10).
• Log-softmax: z_i minus log of the sum of exp of all z_j, which is why log-sum-exp matters.
• The derivative of the natural log is 1/x, and the derivative of exp is exp.

Temperature. Divide scores by T before softmax. With scores 2, 1, 0.1:

• T = 0.5 gives 0.864, 0.117, 0.019 (sharper, more confident).
• T = 1 gives 0.659, 0.242, 0.099.
• T = 2 gives 0.502, 0.304, 0.194 (flatter, more random). As T goes toward 0, the largest score takes all the probability (argmax). As T grows very large, the output tends toward equal shares. Language models use temperature to control how adventurous the next word is.

Variants and relatives.

• Hard max (argmax): picks one winner, but has no useful gradient.
• Log-softmax and cross-entropy fused: computed together for stability.
• Sparsemax and other softmax alternatives: some outputs can be exactly zero (a research topic, mentioned only for context).
• Tanh: a rescaled sigmoid with range minus 1 to 1. In fact tanh(x) = 2σ(2x) − 1.
• Multi-label classification: use an independent sigmoid per label when several labels can be true together, and softmax when exactly one is true.
• Gumbel-softmax: a way to sample while keeping gradients (an advanced topic, background only).

Background (history). The number e appeared in studies of compound interest in the 1600s, and Euler popularised the symbol e later. The logistic curve was introduced to model population growth in the 1800s, and the 'softmax' name was coined much later in the neural-network literature. Treat these as background and not as precise dates. Logarithm tables were a big calculating aid before computers.

Limitations.

• Softmax outputs are not guaranteed to be well calibrated confidence. A network can say 0.99 and be wrong.
• Softmax must pick a winner among known classes, so it cannot say 'none of the above' unless you add such a class.
• Sigmoid and softmax saturate, giving tiny gradients.
• Exp overflows easily. Deep Learning, section 4.1, notes that large inputs become infinity and then NaN, which is why implementations subtract the maximum (see 1.29).

Roadmap links. This topic leans on derivatives (1.25 Jacobian and Hessian) and the Taylor approximation (1.26), since exp and log are the standard Taylor-series examples. The integral from 1.27 defines the natural log as the area under 1/x. After this comes 1.29 Numerical stability and log-sum-exp, then 1.30 Lagrange multipliers and 1.31 Convex sets and functions, where log-sum-exp is shown to be convex. In Module 2, loss functions (2.05) build directly on softmax with cross-entropy, and logistic regression is exactly a sigmoid on a linear score.

6. Worked examples with real numbers

Example A: Tara's softmax, step by step.

1. Scores: 2, 1, 0.1.
2. Exponentials: exp(2) = 7.389, exp(1) = 2.718, exp(0.1) = 1.105.
3. Sum: 7.389 + 2.718 + 1.105 = 11.2125 (using unrounded values).
4. Divide: 7.389 / 11.2125 = 0.659, 2.718 / 11.2125 = 0.242, 1.105 / 11.2125 = 0.099.
5. Check: 0.659 + 0.242 + 0.099 = 1.000. The probabilities sum to 1.
6. Loss if the correct class is the first: minus the natural log of 0.659 is about 0.417. If the correct class were the third: minus the log of 0.099 is about 2.313. The model is punished far more for giving the true answer a small probability.

Example B: sigmoid values and saturation.

• σ(0) = 0.5 and slope 0.25 (the maximum).
• σ(2) = 0.8808 and σ(−2) = 0.1192. They sum to 1.
• σ(5) = 0.9933 and σ(−5) = 0.0067. The slope at 5 is 0.9933 x 0.0067 = 0.0066, about 38 times smaller than at 0.
• σ(10) = 0.99995 and the slope is about 0.000045, essentially zero. A neuron stuck out here barely learns.

Example C: shift invariance and the two-class case.

• softmax(2, 1, 0.1) and softmax(102, 101, 100.1) give the same answer, because adding c = 100 to every score cancels out.
• With scores a = 1.5 and b = 0.2: exp(1.5) = 4.4817 and exp(0.2) = 1.2214, so the first-class probability is 4.4817 / 5.7031 = 0.7858. The sigmoid of (1.5 − 0.2) = sigmoid of 1.3 is also 0.7858.

Example D: logs turn products into sums.

• log10(100) + log10(1000) = 2 + 3 = 5, and log10(100000) = 5.
• Natural log: log(e^3) = 3. If you multiply 1000 probabilities each of 0.5, you get 0.5^1000, about 9.3e−302, which nearly underflows even a 64-bit float, but the log-sum is 1000 x ln(0.5), about −693.1, a perfectly ordinary number.

Example E: e as a limit. Compounding 100 percent interest in n steps gives (1 + 1/n)^n. For n = 1000 the value is 2.7169, close to e = 2.7183. A continuous growth rate of 0.1 over 10 periods multiplies a quantity by exp(1) = 2.718, while ten yearly 10 percent steps give 1.1^10 = 2.594.

7. Comparison tables

Sigmoid, softmax and friends

Function	Input	Output range	Outputs sum to 1?	Typical use
Sigmoid	One real number	Between 0 and 1	Not applicable (one output)	Binary yes or no probability, gates in LSTMs
Softmax	A vector of scores	Each between 0 and 1	Yes	One-of-many classification, attention weights
Tanh	One real number	Between −1 and 1	No	Hidden units, centred outputs
ReLU	One real number	0 and above	No	Hidden layers
Softplus	One real number	Above 0	No	Positive values such as a variance

Effect of temperature on scores 2, 1, 0.1

Temperature	Prob. of top	Second	Third	Feel
0.5	0.864	0.117	0.019	Confident
1	0.659	0.242	0.099	Standard
2	0.502	0.304	0.194	Flat, adventurous

Exp and log as opposites

Feature	exp(x)	log(x)
Domain (allowed inputs)	All real numbers	Positive numbers only
Output	Always positive	Any real number
Turns	Adding into multiplying	Multiplying into adding
Slope	exp(x)	1/x
Value at 0 and 1	exp(0) = 1	log(1) = 0

Sigmoid value ladder

Input x	Sigmoid	Slope
−10	0.00005	0.000045
−5	0.0067	0.0066
−2	0.1192	0.1050
0	0.5	0.25
2	0.8808	0.1050
5	0.9933	0.0066
10	0.99995	0.000045

8. Common mistakes and myths

• Mistake: treating logits as probabilities. Raw scores can be negative or above 1. Only the output of softmax or sigmoid is a probability vector.
• Mistake: applying softmax twice, for example adding a softmax layer and then using a loss that already expects raw logits. The result is a flattened, badly behaved gradient.
• Myth: softmax picks the max. It is a smooth, weighted version; the biggest score gets the biggest share, not all of it.
• Myth: a high softmax probability means the model is right. It only shows relative preference among the classes listed; the model can be confidently wrong.
• Mistake: logs of zero or negatives. log(0) is minus infinity and log of a negative number is undefined, so add a tiny epsilon or work in log space.
• Mistake: mixing log bases. A natural log and a base-10 log differ by a constant factor of about 2.303 (ln 10 = 2.3026).
• Mistake: writing log(a + b) = log a + log b. Only products split: log(ab) = log a + log b.
• Myth: sigmoid is always bad for deep nets. It is not a good hidden-layer activation for very deep stacks, but it is exactly right at the output of a binary classifier.
• Mistake: forgetting that exp overflows. exp(1000) is infinity even in 64-bit floats, so never exponentiate big raw scores without shifting first.
• Mistake: using softmax for multi-label problems. If several answers can be true at once, use independent sigmoids.

9. Real-life and AI uses

• Classification outputs. Image classifiers and language models end in a softmax that turns scores into class probabilities (Dive into Deep Learning, section 4.1, describes softmax as making outputs non-negative and summing to 1).
• Next-word prediction. A language model gives a score to every word in its vocabulary and softmax turns them into chances; temperature then controls creativity.
• Attention. Transformers use softmax to turn similarity scores into weights that add up to 1.
• Logistic regression. A linear score passed through the sigmoid is a probability, and the logit is the linear score. Doctors and bankers read it as log-odds.
• Gates. LSTM and GRU gates use sigmoid as a soft on-off dial between 0 and 1.
• Loss functions. Cross-entropy is the negative log of the probability of the right answer, and log-likelihood is the standard training target.
• Reinforcement learning. Policies often use softmax over action values, with temperature setting how much to explore.
• Everyday life. Earthquake magnitude and decibels use logarithmic scales, and compound interest and population growth follow exponentials.

10. Self-check questions (with answers)

1. What is exp(0), and what is log(1)? Both rules give the identity: exp(0) = 1 and log(1) = 0.
2. What is log(e^3)? 3, because log undoes exp.
3. Why do logs help when multiplying many probabilities? log(ab) = log a + log b, so a long, tiny product becomes a sum of ordinary-sized numbers.
4. What are the two steps of softmax, and why each? Exponentiate (makes everything positive, keeps order, widens gaps) then divide by the total (makes the numbers sum to 1).
5. Compute softmax of 2, 1, 0.1 approximately. About 0.659, 0.242 and 0.099.
6. What is sigmoid(0), and what is sigmoid(2) plus sigmoid(−2)? 0.5, and exactly 1 (0.8808 + 0.1192), since 1 − σ(x) = σ(−x).
7. What is the largest slope of the sigmoid and where? 0.25, at x = 0, because σ'(x) = σ(x)(1 − σ(x)) = 0.5 x 0.5 there.
8. What does 'saturation' mean? At large positive or negative inputs the sigmoid is flat, so its slope is near zero and learning stalls.
9. Why is softmax(z + c) equal to softmax(z)? exp(z + c) = exp(c) exp(z), and exp(c) cancels between the top and bottom.
10. How are sigmoid and two-class softmax related? softmax for the first of two scores a and b equals σ(a − b); for a = 1.5 and b = 0.2 both give 0.7858.

11. Key takeaways, go deeper, and next

Key takeaways

• Exp turns adding into multiplying; log turns multiplying into adding; they undo each other.
• Sigmoid squashes one score into a probability; its slope is σ(1 − σ), at most 0.25.
• Softmax squashes many scores into probabilities that sum to 1: exponentiate then normalise.
• Only differences between scores matter: softmax(z + c) = softmax(z).
• Two-class softmax is a sigmoid of the score difference.
• Cross-entropy is the negative log of the correct probability, so confident mistakes cost a lot.

Go deeper

• Read the 'Useful Properties of Common Functions' section in the Deep Learning book for sigmoid, softplus and logit identities.
• Code softmax in NumPy and see what happens with scores such as 1000, 999, 998, then fix it in 1.29.
• Plot softmax at different temperatures.
• Derive the gradient of cross-entropy with softmax and notice it is predicted minus true.

Next on the roadmap: 1.29 Numerical stability and log-sum-exp.

12. Sources and further reading

• Deep Learning (Goodfellow, Bengio, Courville, 2016), chapter 3 'Probability and Information Theory', section 3.10 'Useful Properties of Common Functions' (sigmoid, softplus, logit, identities): https://www.deeplearningbook.org
• Deep Learning (Goodfellow, Bengio, Courville, 2016), chapter 4 'Numerical Computation', section 4.1 'Overflow and Underflow' (stabilising softmax by subtracting the maximum): https://www.deeplearningbook.org
• Dive into Deep Learning (d2l.ai), section 4.1 'Softmax Regression': https://d2l.ai/chapter_linear-classification/softmax-regression.html. Outputs become non-negative and sum to 1; log-likelihood and cross-entropy.
• Dive into Deep Learning (d2l.ai), section 4.5 'Concise Implementation of Softmax Regression', subsection 'Softmax Revisited': https://d2l.ai/chapter_linear-classification/softmax-regression-concise.html. Overflow and underflow, subtracting the max, and combining softmax with cross-entropy.
• PyTorch documentation, torch.logsumexp ('The computation is numerically stabilized'): https://docs.pytorch.org/docs/2.3/generated/torch.logsumexp.html

Accuracy note: The softmax, sigmoid, temperature and limit numbers were checked in Python on 2026-11-03; the history paragraph is general background, not sourced dates, and the multi-label and attention uses are standard background.