Measuring Similarity from Embeddings

You now have embeddings for any pair of examples. A similarity measure takes these embeddings and returns a number measuring their similarity. Remember that embeddings are simply vectors of numbers. To find the similarity between two vectors \(A = [a_1,a_2,...,a_n]\) and \(B = [b_1,b_2,...,b_n]\), you have three similarity measures to choose from, as listed in the table below.

MeasureMeaningFormulaRelationship to increasing similarity
Euclidean distanceDistance between ends of vectors \(\sqrt{(a_1-b_1)^2+(a_2-b_2)^2+...+(a_N-b_N)^2}\) Decreases
CosineCosine of angle \(\theta\) between vectors \(\frac{a^T b}{|a| \cdot |b|}\) Increases
Dot ProductCosine multiplied by lengths of both vectors \(a_1b_1+a_2b_2+...+a_nb_n\) \(=|a||b|cos(\theta)\) Increases. Also increases with length of vectors.

Choosing a Similarity Measure

In contrast to the cosine, the dot product is proportional to the vector length. This is important because examples that appear very frequently in the training set (for example, popular YouTube videos) tend to have embedding vectors with large lengths. If you want to capture popularity, then choose dot product. However, the risk is that popular examples may skew the similarity metric. To balance this skew, you can raise the length to an exponent \(\alpha\ < 1\) to calculate the dot product as \(|a|^{\alpha}|b|^{\alpha}\cos(\theta)\).

To better understand how vector length changes the similarity measure, normalize the vector lengths to 1 and notice that the three measures become proportional to each other.

Proof: Proportionality of Similarity Measures
After normalizing a and b such that \(||a||=1\) and \(||b||=1\), these three measures are related as:
  • Euclidean distance = \(||a-b|| = \sqrt{||a||^2 + ||b||^2 - 2a^{T}b} = \sqrt{2-2\cos(\theta_{ab})}\).
  • Dot product = \( |a||b| \cos(\theta_{ab}) = 1\cdot1\cdot \cos(\theta_{ab}) = cos(\theta_{ab})\).
  • Cosine = \(\cos(\theta_{ab})\).
Thus, all three similarity measures are equivalent because they are proportional to \(cos(\theta_{ab})\).