1.04 Dot product and cosine similarity: study notes

Learn AI by Story

1. The idea in one sentence

The dot product multiplies two lists of numbers pair by pair and adds the results into one score, and cosine similarity divides that score by the lengths of the lists so it measures only how closely the two point in the same direction, whatever their sizes.

2. The story from the video

Asha compares songs as vectors of [action, calm]. Song A is [4, 1], B is [3, 2] and C is [1, 4]. The dot product of A and B is 4×3 + 1×2 = 14, while A and C give 4×1 + 1×4 = 8, so B is more alike A. Drawn as arrows, A and B have a small angle between them, and A and C a wide one. But there is a catch: double B to [6, 4] and the score jumps to 28 although the direction has not changed. Cosine similarity fixes this by dividing by the lengths, giving 0.94 for both B and doubled B. It is the tool behind search, recommendations, embeddings and RAG.

3. Key definitions (glossary)

Term	Precise definition	Plain English
Vector	An ordered list of numbers that can be drawn as an arrow from the origin.	A list of numbers, or an arrow.
Component	One number in a vector.	One entry, for example the "action" score.
Dot product (algebraic)	For a = [a1, ..., an] and b = [b1, ..., bn], a · b = a1×b1 + ... + an×bn.	Multiply pairs, add.
Dot product (geometric)	a · b = ‖a‖ ‖b‖ cos(θ), where θ is the angle between them.	Lengths times how aligned they are.
Magnitude (length, norm)	‖a‖ = sqrt(a1² + ... + an²), the Euclidean (L2) norm.	How long the arrow is.
Unit vector	A vector of length 1.	An arrow of standard length.
Normalisation	Dividing a vector by its length so it becomes a unit vector.	Shrink or stretch to length 1, keeping direction.
Angle θ	The angle between two arrows placed tail to tail.	How far apart the directions are.
Cosine	For an angle θ, the ratio that is 1 at 0°, 0 at 90° and -1 at 180°.	A smooth "alignment meter".
Cosine similarity	cos(θ) = (a · b) / (‖a‖ ‖b‖).	Dot product with size removed.
Cosine distance	1 − cosine similarity.	Turns "alike" into "far apart".
Euclidean distance	The straight-line distance sqrt of the sum of squared differences between two points.	A ruler measurement between the arrow tips.
Orthogonal	Two vectors with dot product 0 (angle 90°).	Perpendicular, nothing in common.
Projection	The part of a that lies along b: ((a · b) / (b · b)) b.	The shadow of one arrow on another.
Scalar	A single number.	One number, not a list.
Embedding	A vector of numbers that represents a word, sentence, image or song so that similar things are close.	A numeric fingerprint of meaning.
Nearest-neighbour search	Finding the stored vectors most similar to a query vector.	Find the closest matches.
Vector database	A system that stores embeddings and answers nearest-neighbour queries quickly.	A search engine for fingerprints.
Bag of words	Representing a text by how many times each word occurs, ignoring order.	A word tally.
TF-IDF	A weighting that boosts words frequent in one document and rare overall.	Counts that care about rare, telling words.

4. Step-by-step explanation

Step 1: Turn things into vectors

Choose features and give each item a number per feature. Songs become [action, calm]. A, B and C become [4, 1], [3, 2], [1, 4]. Why: numbers can be compared mechanically; a song title cannot.

Step 2: Multiply matching components and add

A · B = 4×3 + 1×2 = 12 + 2 = 14. A · C = 4×1 + 1×4 = 4 + 4 = 8. Why: when both vectors are big in the same feature, that product is big, so the sum rewards agreement. If one is big where the other is small, the product is modest.

Step 3: Read the scores

14 is larger than 8, so A is more similar to B than to C. Why: B has high action and some calm like A. C is mainly calm.

Step 4: See the picture

Draw A, B and C as arrows. A points mostly right (action) with a little up (calm). B points right and a bit more up. C points mostly up. The angle between A and B is about 19.7 degrees; between A and C about 61.9 degrees. Why: the dot product secretly depends on this angle, which is why it rewards arrows that point the same way.

Step 5: Spot the catch

Double B to B' = [6, 4]. Its direction has not changed; it is just louder. But A · B' = 4×6 + 1×4 = 28, double the old score. Why it matters: if one song is simply "more of everything", or one document is simply longer, the raw dot product rewards size rather than likeness.

Step 6: Divide out the lengths

‖A‖ = sqrt(4² + 1²) = sqrt(17) = 4.123. ‖B‖ = sqrt(3² + 2²) = sqrt(13) = 3.606. Then cosine similarity = 14 / (4.123 × 3.606) = 14 / 14.866 = 0.94. For B', ‖B'‖ = sqrt(36 + 16) = sqrt(52) = 7.211, and 28 / (4.123 × 7.211) = 28 / 29.73 = 0.94. Same answer. Why: dividing by the lengths cancels any stretching, leaving only direction.

Step 7: Read the final score

+1 means the same direction, 0 means a right angle (unrelated), and -1 means exactly opposite. For A and C: 8 / (sqrt(17) × sqrt(17)) = 8 / 17 = 0.47. So A and B (0.94) are much closer in taste than A and C (0.47).

5. Everything the video skipped

Two definitions of the dot product, and why they agree

Algebraic: a · b = a1×b1 + a2×b2 + ... (what you compute).
Geometric: a · b = ‖a‖ ‖b‖ cos(θ) (what it means).

They agree because the length formula and the law of cosines (a triangle rule that relates side lengths to an angle) force the same number. In two dimensions, write a = ‖a‖(cos α, sin α) and b = ‖b‖(cos β, sin β). Then a · b = ‖a‖‖b‖(cos α cos β + sin α sin β) = ‖a‖‖b‖ cos(α − β). The angle between them is α − β. Check with the song example: A at 14.04 degrees, B at 33.69 degrees, the gap is 19.65 degrees, and cos(19.65°) = 0.9417, matching 14 / 14.866. Each definition is useful: the first to compute, the second to understand.

Dot product and length

A vector dotted with itself gives its squared length: a · a = a1² + a2² + ... = ‖a‖². For A = [4, 1]: 16 + 1 = 17, so ‖A‖ = sqrt(17). That is why cosine similarity of a vector with itself is always 1: (a · a) / (‖a‖‖a‖) = 1.

Angle recovery

Because cos(θ) = (a · b) / (‖a‖‖b‖), you can recover the angle with the inverse cosine: θ = arccos(0.9417) = 19.65 degrees for A and B, and θ = arccos(0.4706) = 61.93 degrees for A and C. In machine learning one usually keeps the cosine and skips the angle.

Orthogonality

If a · b = 0 the vectors are orthogonal (perpendicular). Example: [2, 1] · [-1, 2] = -2 + 2 = 0. In features terms, nothing in one vector's pattern lines up with the other. Orthogonal does not mean "opposite"; opposite would be cosine -1, for example [2, 1] and [-2, -1].

Projection

How much of a points along b? The scalar projection is (a · b) / ‖b‖. For A on B: 14 / 3.606 = 3.88. The projection vector is ((a · b)/(b · b)) b = (14/13) × [3, 2] = [3.23, 2.15]. Think of the shadow of A when light shines at right angles to B. This idea underlies many methods, including regression and principal component analysis (PCA).

Cosine distance

Distance = 1 − similarity. For A and B: 1 − 0.9417 = 0.058. For A and C: 1 − 0.4706 = 0.529. Nearest neighbours are the ones with smallest distance, equivalent to largest similarity. Strictly, cosine distance is not a true metric (it can break the triangle inequality), but it ranks results well.

Euclidean distance versus cosine

Pair	Dot product	Cosine	Euclidean distance
A, B = [3, 2]	14	0.942	sqrt(1 + 1) = 1.414
A, B' = [6, 4]	28	0.942	sqrt(4 + 9) = 3.606
A, C = [1, 4]	8	0.471	sqrt(9 + 9) = 4.243

Euclidean distance says B' is far from A (3.606) even though it points exactly like B. Cosine says they are equally similar. Choose cosine when direction (the pattern) matters and size does not, such as text of different lengths. Choose Euclidean when size matters, such as physical positions.

Normalised vectors

If every vector is first scaled to length 1, the lengths in the denominator are 1, so cosine similarity equals the plain dot product. Check: A / ‖A‖ = [0.970, 0.243] and B / ‖B‖ = [0.832, 0.555]. Their dot product is 0.807 + 0.135 = 0.942. This is why search systems often store normalised vectors: one fast dot product gives cosine.

Dot product and matrices

A matrix product (1.03) is a table of dot products: each cell is the dot product of a row with a column. So computing similarity between 1,000 queries and 1,000,000 stored items is just one huge matrix multiplication, which GPUs do well.

Attention in transformers (preview)

Transformers (a later module) let each word look at other words. How much word i attends to word j starts as the dot product of a "query" vector for i and a "key" vector for j. The scores are divided by sqrt(d), where d is the vector size, to stop them growing too large (an illustrative d of 64 means dividing by 8), then converted to weights. So a scaled dot product is the core of modern language models.

Embeddings, nearest neighbours and vector databases

An embedding model turns text, images or songs into vectors where similar meaning gives similar direction. To search, you embed the query, then find the stored vectors with the highest cosine similarity. Doing this exactly over millions of items is slow, so vector databases use approximate nearest-neighbour indexes to trade a little accuracy for speed. This is the retrieval step in RAG (retrieval-augmented generation): fetch the passages closest to the question, then give them to a language model.

Negative cosine in embeddings

In theory cosine runs from -1 to 1. In many text embeddings, scores for unrelated texts sit near 0 and truly opposite meaning rarely gives exactly -1; the geometry is not simply "opposite equals negative". Do not read a negative value as a guaranteed antonym. Count-based vectors (word counts) never go negative, so their cosine runs only from 0 to 1.

Limitations

Magnitude is thrown away. Sometimes size carries information, such as how strongly someone likes a genre. Then dot product or Euclidean distance may fit better, or add magnitude as a separate feature.
Zero vectors have no direction; cosine is undefined (division by zero).
High dimensions: with hundreds or thousands of dimensions, random vectors tend to be nearly orthogonal and distances bunch together, so small differences in scores can be noise.
Quality of embeddings: cosine can only be as good as the vectors. A poor embedding gives confident but wrong matches.
Only linear alignment: it compares patterns across features but not their order or deeper logic.
Common words: in counts, words like "the" can dominate; stop-word removal or TF-IDF helps.

History in two lines

The dot product emerged from 19th-century vector work by Hamilton, Gibbs and Heaviside. Cosine similarity became central to information retrieval in the 1960s to 1970s with the vector space model of documents by Gerard Salton.

Where it sits on the roadmap

Builds on 1.01 Vectors and 1.03 Matrix multiplication (each cell is a dot product).
Previews attention, embeddings and retrieval in later modules.
Next, 1.05 Functions and slope begins calculus, the tools for learning by small adjustments.

6. Worked examples

Example 1: the three songs

A = [4, 1], B = [3, 2], C = [1, 4].

A · B = 4×3 + 1×2 = 14. A · C = 4×1 + 1×4 = 8. B · C = 3×1 + 2×4 = 11.
Lengths: ‖A‖ = sqrt(17) = 4.123, ‖B‖ = sqrt(13) = 3.606, ‖C‖ = sqrt(17) = 4.123.
Cosines: A, B = 14 / (4.123 × 3.606) = 14 / 14.866 = 0.942. A, C = 8 / 17 = 0.471. B, C = 11 / (3.606 × 4.123) = 11 / 14.866 = 0.740.
Doubling B: B' = [6, 4]. A · B' = 28. ‖B'‖ = sqrt(52) = 7.211. Cosine = 28 / (4.123 × 7.211) = 28 / 29.73 = 0.942, unchanged.

Ranking by cosine: A-B (0.94) is the closest pair, B-C (0.74) next, A-C (0.47) the farthest. Ranking by raw dot product would put A-B' (28) far above everything just because B' is large.

Example 2: comparing short documents with word counts

Documents:

D1: "the cat sat on the mat"
D2: "the cat sat on the sofa"
D3: "the dog chased the ball"

Vocabulary in order: the, cat, sat, on, mat, sofa, dog, chased, ball.

Doc	the	cat	sat	on	mat	sofa	dog	chased	ball
D1	2	1	1	1	1	0	0	0	0
D2	2	1	1	1	0	1	0	0	0
D3	2	0	0	0	0	0	1	1	1

D1 · D2 = 2×2 + 1×1 + 1×1 + 1×1 + 0 + 0 + 0 + 0 + 0 = 4 + 1 + 1 + 1 = 7. ‖D1‖ = sqrt(4 + 1 + 1 + 1 + 1) = sqrt(8) = 2.828. ‖D2‖ = sqrt(4 + 1 + 1 + 1 + 1) = sqrt(8) = 2.828. Cosine(D1, D2) = 7 / 8 = 0.875.

D1 · D3 = 2×2 = 4 (only "the" is shared). ‖D3‖ = sqrt(4 + 1 + 1 + 1) = sqrt(7) = 2.646. Cosine(D1, D3) = 4 / (2.828 × 2.646) = 4 / 7.483 = 0.535.

D1 and D2 are clearly more alike (0.875 versus 0.535). But 0.535 is surprisingly high for sentences about different things: the only shared word is "the". Remove the common words "the" and "on" and the counts of D1 become cat, sat, mat = 1 each, D3 become dog, chased, ball = 1 each; the dot product is 0 and the cosine is 0. This shows why stop-word removal and TF-IDF matter.

Length independence: if D1 is pasted twice ("the cat sat on the mat the cat sat on the mat") its counts all double, the dot product with D2 doubles to 14, and the cosine stays 14 / (5.657 × 2.828) = 14 / 16 = 0.875.

Example 3: orthogonal and opposite vectors

u = [2, 1], v = [-1, 2]: u · v = -2 + 2 = 0, cosine 0 (right angle).
u and w = [-2, -1]: u · w = -4 - 1 = -5; ‖u‖ = ‖w‖ = sqrt(5), so cosine = -5 / 5 = -1 (opposite).

7. Comparison table

Measure	Formula idea	Sees size?	Range	Best when
Dot product	Multiply pairs, add	Yes	Any number	Size is meaningful; normalised vectors
Cosine similarity	Dot product ÷ (length × length)	No	-1 to 1	Direction or pattern matters
Cosine distance	1 − cosine	No	0 to 2	You need a "distance" for search
Euclidean distance	Straight-line gap of tips	Yes	0 and up	Position and size matter

Reading cosine scores:

Cosine	Angle (approx.)	Meaning
1	0°	Same direction
0.94	20°	Very alike (A and B)
0.47	62°	Somewhat alike (A and C)
0	90°	Unrelated
-1	180°	Opposite

8. Common mistakes and myths

Myth: "A bigger dot product always means more similar." Not if the vectors have different lengths: B' scored 28 only because it is twice as long.
Mistake: forgetting to divide by both lengths. The denominator is ‖A‖ times ‖B‖, not one of them.
Mistake: taking a square root of the wrong thing. Length is the square root of the sum of squares, not the sum of the entries.
Mistake: mismatched lengths. You cannot dot a 2-number vector with a 3-number one.
Myth: "Cosine 0 means opposite." Zero means unrelated (perpendicular). Opposite is -1.
Myth: "Cosine similarity is a percentage." 0.94 does not mean 94% the same; it is the cosine of an angle.
Mistake: using cosine on a zero vector. It has no direction and gives a division by zero.
Myth: "Embeddings with cosine 0.8 always mean related." Typical scores depend on the model; compare scores within the same model.
Mistake: mixing up distance and similarity. Larger similarity means closer; larger distance means farther.

9. Real-life uses with concrete examples

Music and video recommendations: Each song or viewer is a vector of tastes; show the songs with the highest cosine to what you already like.
Web and document search: The query and each page become vectors; results are sorted by cosine similarity.
Semantic search with embeddings: Searching "cheap places to sleep" finds "budget hotels" because their embeddings point the same way.
RAG chatbots: A question is embedded, the closest passages from company documents are retrieved by cosine, and the model answers from them.
Duplicate and plagiarism detection: Two texts with cosine near 1 are probably near copies.
Face and image matching: Two face embeddings with high cosine likely show the same person.
Transformers: Every attention head computes scaled dot products between queries and keys.
Physics and graphics: The dot product gives work done by a force, and light on a surface depends on the dot product between the surface direction and the light direction.

10. Self-check questions

Compute [2, 3] · [4, 1].
What does the dot product of a vector with itself give?
Why can the raw dot product mislead when comparing items of different sizes?
Compute the cosine similarity of [1, 0] and [1, 1].
What does a cosine similarity of 0 mean? Of -1?
If you double one vector, what happens to the dot product and to the cosine similarity?
What is cosine distance, and what is it for A and B in the songs example?
If both vectors are normalised to length 1, what is cosine similarity?
Give two limitations of cosine similarity.
Where does the dot product appear inside a transformer?

Answers

2×4 + 3×1 = 8 + 3 = 11.
Its squared length: a · a = ‖a‖².
The score grows with length. A long song, document or loud vector scores high even if its direction is no more similar.
Dot = 1×1 + 0×1 = 1. Lengths: 1 and sqrt(2) = 1.414. Cosine = 1 / 1.414 = 0.707 (an angle of 45 degrees).
0 means perpendicular, with no linear relation; -1 means exactly opposite directions.
The dot product doubles; the cosine similarity stays the same.
1 minus cosine similarity. For A and B: 1 − 0.942 = 0.058.
It equals the plain dot product, because both lengths are 1.
For example: it ignores magnitude; it is undefined for a zero vector; in very high dimensions scores bunch together; it depends on embedding quality.
In attention, where the dot product of a query and a key (divided by the square root of the vector size) gives the attention score.

11. Key takeaways, go-deeper list, and what comes next

Key takeaways

Dot product = multiply pairs and add; geometrically ‖a‖‖b‖cos(θ).
It rewards alignment, but also size, so raw scores can mislead.
Cosine similarity = dot ÷ (‖a‖‖b‖): 1 same direction, 0 unrelated, -1 opposite.
With unit vectors, cosine equals the dot product, and a matrix product computes many at once.
It powers search, recommendations, embeddings, RAG and attention. Its blind spot is magnitude.

Go deeper

Recompute the song cosines with a third feature such as "lyrics" to see how a third dimension changes the angles.
Try the document example with TF-IDF weights instead of raw counts.
Compare cosine and Euclidean nearest neighbours on a handful of points you draw.
Read about approximate nearest-neighbour methods such as HNSW and about vector databases.
Preview self-attention in the transformer module.

Next roadmap ID: 1.05 Functions and slope (a derivative preview). You have learned how to measure similarity between things; next you will learn how a function changes as its input changes, the first step toward how models learn.

Dot product and cosine similarity: study notes